United States
FAST DELIVERY ∙ EXPERT TECH SUPPORT
UK STOCK ∙ 2 YEAR WARRANTY
Eversolo DMP-A6
Digital Audio Streamer with ESS DAC.
  INCLUDES EXPRESS 2-4 DAY DELIVERY 

Author Topic: Tutorial: Using HDD with sizes > 2 TB up to 16 TB and SDHC/SDXC Cards  (Read 25422 times)

Nice Monkey

  • Senior Member
  • ***
  • Posts: 1128
  • Loves classic movies
    • View Profile
Quick short advice: Use a W10/W11 PC with the target HDD (E)SATA attached (may be a problem to find one):
- Be sure to remove all partions including any hidden partitions first
- Format a single partition with defaults using GPT + NTFS
Never found one that did not work next up to 16 TByte.

Part-1

OK finally figured out how that limit of > 2TB HDD support actually works and what can be done and what can not be done to overcome limitations.
Everything mentioned here was implemented by myself in practice. Practical experience includes using 2, 3, 4, 5, 6, 8, 10, 12 and 16 TB drives.

After reading this tutorial you will see that HDD > 2TB is supported both as Internal HDD and as USB attached HDD. There are however conditions to this and formats under both conditions in fact are (mostly) conflictive with each other.

I am afraid this is a bit technical, but as many of us including support staff was often confused about things, I will try to make it clear(er). If you are only interested in the do's and don'ts to make it work you may jump straight to Part-2.

As a start one needs to understand the difference between "HDD Initiation" and "HDD Formatting".
Often this is done in one step by a program, but the technical functions are different.
- The "HDD Initiation" tells the interface driver where and how to Boot, how the HDD can physically be accessed and where the physical partitions can be found on the disc. Here Microsoft Windows MBR or GPT initiations will be used to be compatibel with various Media Players
- "HDD Formatting" puts on the filesystem. This defines how and where files are put on the disk using as specific filesystem. Here the filesystem is assumed always to be Microsoft Windows NTFS to remain compatible with most Media Players, FAT32 works functionally but has a 4 GByte individual file size upper limit. That won't do for bigger MKV files and above all ISO images of DVD and BD. BD in fact will hardly ever work.

Some technical background for the interested
- First there are the Hard Drive manufacturers which make HDD drives traditonally read by sectors of 512 bytes each. With modern discs in Terabytes that is a lot of sectors which is not very efficient any more. Filesystems like FAT32 and NTFS therefore always read/write bigger blocks of data called clusters. A cluster is by default 4 KBytes (=8 sectors) and is configurable.

- Sectors are referenced to using 32-bit registers. If one calculates right one learns that a maximum 2.2 TByte can be addressed with a 32- bit counter adressing 512 byte sectors. Changing that is far from simple as this resides deep in the OS and Drivers. Partioning does not help here as still the whole disc space needs to be accessed at the physical level. This was up till recent done via the Master Boot Record (MBR) and Logical Block Addressing (LBA) using 32-bit. Windows changed to GPT (GUID Partion Table) with Windows-7 using 64-bit addressing solving this limitation structurally.

- But even with GPT the read/write overhead remains the same. So Hardware manufactures did a trick and intruduced Physical and Logical sectors. The newer 2 Tera Byte and bigger discs all work with Physical Sectors of 4K Bytes presented to the software as 8 Logical sectors of 512 bytes. This technology is called "Advanced Format 4K" which reduces the disc internal overhead but not the total CPU processing chain for I/O. The drives should be labeled 512e which stands for 512 byte logical sector emulation. To make a 512e HDD efficient sectors must be aligned at 4K boundaries. See the matching AF logo for a 512e HDD.

- The next step is Native 4K support. See attached logo for these 4Kn HDD. With 4Kn drives Logical Sectors, Physical Sectors and Clusters are all identical 4K Bytes streamlining throughput that way. This requires drivers dealing with 4K Logical sectors and obviously bolts on support of GPT for Initiation. Windows supports Native 4K HDD starting with Windows 8.1 OS. This paves the road for hassle-free support of > 2 TB and even > 16 TB HDD combined with improved performance. Starting begin 2016 the first Native 4K HDD's made by WD appeared on the market. These are sold as HDD or build-in into their USB3 Enclosures. Drives can easily be identified as both Logical Sectors and Physical Sectors are always 4K.

You can display all mentioned HDD format details using fsutil fsinfo ntfsinfo x: under Windows. This is a CMD to be executed with Admin rights! (Note x: replaces the actual drive being used)
You may also use "MiniTool Partition Wizard Free" (see bottom for more details).

Back to most media players around
Access can run via Internal HDD, USB HDD Docking Bay, USB HDD enclosure or a NAS.
- A NAS setup hides the total physical drive access plus format (RAID) used and as such supports > 2TB drives without the Media Player even being aware of it.

- Internal discs are SATA attached (like internal drives in your PC). Drivers for SATA are easier to adapt and most media players upgraded to GPT support. They can address > 2TB discs using Advanced Format 4K support. All external drives for mounting into a PC purchased in shops are formatted that way. There is always a buffering/memory limitation somewhere but there is not a hard limitation for the maximum disc size. We know that 6 TB and 8 TB drives do work, but I do expect drives far above 16 TB to work too.

- USB attached drives is a complete different ballgame. Drivers are in all places and often very old. Also 64-bit addressing support may be jeopardized by a structural HW limitation. So GPT support is often not yet available and worse not being planned for many products. Under the condition that GPT is handled by products on both ends then USB standards do not impose limitations. In practice only some recent USB3 Docking Bay implementations do include such support.
As a temporary fix hardware manufactures came up with a trick. They started to ship USB-enclosures (= Casing, USB/ATA bridge and a HDD as one unit) with native 4K Logical sector enabled. The USB/ATA bridge firmware presents these to the USB-side. Result MBR can now work with 32-bit adressing of 4K Logical sectors. The physical disk access is not a problem as this is done by the USB/ATA bridge here. This trick will obviously only work up to a limit of just above 16 TB capacity for older USB/ATA bridges but recent versions may support 64-bit LBA.

Part-2

Moving a HDD between Internal <=> External <=> NAS <=> Windows PC
Please use Microsoft Windows 10/11 only for re-formatting with GPT+NTFS and have it (E)SATA attached for Internal HDD's. I am sure that will work. MAC and Linux equivalents may do the trick, but never tried those myself. Nobody ever reported (good) results for media players
Internal HDD and USB HDD > 2TB are not inter-changeable and need to be re-initiated and re-formatted for each deployment. As most people don't open USB Drives/Housings/Enclosures they will never be aware of the difference. Personally I am now able to move any HDD in either direction without problems. This with the exception of (still exotic) 4Kn type HDD's having 4K Physical sectors not handled by Android players.

- A HDD pulled from a USB Drive/Housing/Enclosure will work only as an internal SATA attached drive again after standard re-initialisation + re-formatting. Obviously this HDD must be (E)SATA attached during mandatory re-initialisation + re-formatting using Windows-7 or equivalent standard tools. A common reason for doing so is that a complete USB-Drive may factually be cheaper than buying the same HDD as a separate unit. You still may be able to re-use the then empty USB-External enclosure later.

- A HDD pulled from a Plug & Play USB3 Docking Bay with GPT support (like Fantec's, Icy Box and probably Lindy's) will work straight as an Internal HDD again. In fact you may move individual units freely between such a Docking Bay and Internal bays of multiple brands. Tried this with sizes up to 8 TB. This won't work with RAID supporting variants obviously as these will need a re-build of many hours when a HDD is changed (even in JBOD mode).

- Formatting with 4K Logical sectors for USB External drives requires special tooling, it also helps if you exactly know what you are doing using such tools. I am using tooling from Acronis for this just as most USB HDD manufacturers do. You will run into this whenever you want to insert a new/re-used standard SATA internal HDD into an existing External USB Housing e.g. to increase its capacity.

- If you pull a HDD from a NAS then it won't work for either deployment as it has been re-formatted the wrong way. It may be using EXT3/EXT4 but in most cases will be using a proprietary format. Re-format it first for the purpose you have in mind being Internal or External USB enclosure as explained.

- A HDD pulled from a working Windows PC might still not work despite being formatted correctly with GPT + NTFS. The reason is Windows and/or the manufacturer has put an additional (hidden) partition on it for OS recovery purposes. Media players like to see a single Partition starting at the beginning of the physical disk. You need to remove all existing partitions first and next format again with GPT + NTFS when (E)SATA attached. Doing so via USB won't work in most cases neither as resulting HDD parameters are slightly different also not interpreted by Android.

Using (EXT2/EXT3) EXT4 formatted HDD in combination with MAC or Unix/Linux/Android based platforms.
First of all this is not officially supported by some media players but it is validated to be working just fine. This does not come as a real surprise as Android is a Linux derivation and Android kernels include support for EXT2/3/4.
EXT2 is often used on USB-sticks and SD-cards, but is less suitable (=less robust) for deployment on a HDD. Instead use EXT4 by preference.

Obviously EXT formatting has to deal with the same HDD technical developments as NTFS:
- EXT3 will support a HDD up to 16 TB using a Logical 4K block size The choice of the block size to be used may be automatic or a variable to be set manually; this depending on the OS, OS-version and the factual tool being used.
- EXT4 has no implicit HDD upper limit regarding total volume. Per OS and OS-version a support limit may be applicable and specified by the manufacturer. In practice EXT4 will work with any size HDD available within the foreseeable future.

SDHC/SDXC Cards and using an exFAT for HDD?
exFAT is the new defacto File System standard for SDXC cards. In fact exFAT was designed for memory cards and sticks and not for HDD. It is an excellent replacement for FAT32 removing all its practical limitations but not an advisable (=less robust) replacement for NTFS.
As SD-Cards are immensely popular this will move exFAT to almost any platform. You see already a lot of devices (above all phones) with SDXC/exFAT support included. Media Players are very slow to adopt SDXC card support. Using exFAT on SDHC cards should work in normal cases as it is a standard Android Lollipop feature by now.

Occasionally I read that only exFAT support makes the difference between SDHC and SDXC, this is a mistake as support for the new UHS-I bus is the real technical innovation coming with SDXC. This new SDXC bus has a far higher speed . The UHS-I backward compatible UHS-II bus will again double throughput and is almost as fast as SATA600 and has 2 rows of pins making identification easy. Some brands call everything up to 32 GByte SDHC and above 32 SDXC which is misleading also. I have seen 128 GByte SDHC cards not supporting the UHS-I bus. Look at the corresponding UHS-I/UHS-II speed labels which must be present for SDXC compatibility and performance indication. Media players will not work with native SDXC cards! Never seen any UHS-I bus enabled media players yet which is a pity.

SDHC/SDXC Hybrid Cards which also exist should always carry both a SDHC (e.g. @10) and SDXC (e.g. U1) speed indicator as they work when inserted in old SDHC slots and new SDXC slots, but in fact may not always carry both labels. SANdisk is known to make a real mess with labeling. See the attachment for a correctly labeled Kingston card. The term hybrid card which I use here is never used for any SD-card advertising which adds to the confusion.
When you e.g. put a 32 GByte Hybrid card (mostly labeled as SDHC) into a card-reader for SDXC with UHS support, then it will be blazing fast. The reverse is also true, putting a Hybrid card of 64+ GByte into an SDHC card-reader slot will slow it down to @10 speeds maximum even being an ultra fast UHS-II card. This will happen when using such a hybrid card with media players always.
When using a SDXC capable external USB Card Reader then it must be USB3 connected as transfer speeds are way above USB2 limits. Measured 85 MB/sec Read and 30 MB/sec Write on a 12 Euro 32 GB Hybrid card. This card works also with most of my media players. One player (Xtreamer eXpress) refuses to read hybrid cards for an unknown reason.

If you want to validate/view very detailed HDD settings yourself and/or to try various formats suitable for Media Players then you can use the excellent Windows based "MiniTool Partition Wizard Free".
https://www.partitionwizard.com/free-partition-manager.html
It allows HDD formatting with EXT2/EXT3/EXT4 using Windows too


Using Android based media players with SATA attached disks > 16 TByte
Android uses a 32-bit LBA register to address Physical Sectors sectors causing the infamous HDD 2,2 TByte limit with 512 byte sectors. Using the 512e trick one can access just over 17,6 TByte with 4K Physical sectors emulating 8x 512 byte Logical Sectors hence 512e format.
Will be glad somebody convincing me being wrong here.

The unrelated cluster size is how big the OS writes the data single I/O in multiple logical sectors. Up to 32K allowed but may increase slack with many small files on it but also reducing I/O overhead with big/huge files like ISO's.

I have heard of persons being able to format 18 TByte and 20 TByte HDD's with NTFS and also claims with those drives to be 512e compatible but never working them on Android without problems next.

Using a 32-bit LBA counter with 4K Physical Sectors results in a 16 Tbyte GPT limit as far as I know. W10 64-bit and recent USB drives both support the essential 64-bit LBA adressing for new 18 ExaByte GPT total HDD limits. W10-64bit seems to allow 256 TB per partition with 128 partitions maximum not causing any practical limits for the foreseeable future.

Practical testing with Zidoo media players and a Seagate EXOS 18 TByte drive showed the limit to be just over 17.5 TByte (=16300 Gigabyte). Exactly as calculated by me. I do expect it to apply also to other drives over 16 TByte and all Android based media players.

Now my message seems to be confirmed at least by Dune:
Latest FW update 221220_0257_r21 release note:
- All models: Improvement: HDD format: HDD formatting software updated to a newer version for better compatibility
- All models: Improvement: HDD format: When attempting to format a HDD bigger than 16TB on a 32-bit OS (which is not supported), now a reasonable error message is shown.

You are warned!
« Last Edit: December 22, 2022, 10:13:12 AM by Nice Monkey »

futeko.com

  • Administrator
  • Senior Member
  • *****
  • Posts: 4627
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or via USB enclosure)
« Reply #1 on: February 04, 2016, 02:56:04 PM »
Excellent tutorial. Thanks.

Nice Monkey

  • Senior Member
  • ***
  • Posts: 1128
  • Loves classic movies
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or via USB enclosure)
« Reply #2 on: February 22, 2016, 08:08:23 AM »
Warning: Western Digital recently also ships both 1 and 2 TB Internal and USB External HDD enclosures with 512e. I am a bit puzzled why they decided to do so. Could be for performance reasons or  just to simplify (standardize) their production? Anyway it will confuse owners of media players even more. On PC' s the problem is virtually gone as all old OS having problems are obsolete now and most Windows OS have been upgraded to W10 by now.

Toshiba is shipping Native 4K drives (4Kn). Did not get any of these to work with Egreat A10 players SATA. As these use the same SOC as Himedia I doubt they will work with Q10 Pro. Dealing with the Physical 4K sectors seem to cause the incompatibility. They do work via USB though.

Use the display methods documented to reveal if you are running into this problem.
« Last Edit: February 09, 2022, 11:50:28 AM by Nice Monkey »

Nice Monkey

  • Senior Member
  • ***
  • Posts: 1128
  • Loves classic movies
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or via USB enclosure)
« Reply #3 on: March 29, 2017, 08:00:04 AM »
Tutorial updated on several points. The main change is a negative advice for new HDD's with Native 4K Physical Sectors (4Kn) as these don't work SATA attached with the Hisillicon SOC/SDK used by Himedia and Egreat players. Maybe a driver update can solve this limitation but it can also be a HW limitation by the chipset itself. Older motherboards and drivers in PC's won't work with these 4Kn type HDD neither.
« Last Edit: March 29, 2017, 08:03:40 AM by Nice Monkey »

Nice Monkey

  • Senior Member
  • ***
  • Posts: 1128
  • Loves classic movies
    • View Profile
Updated the post with SDHC/SDXC and Hybrid SD-card support now that people start to use these also for Internal Android storage expansion.
Overall  textual update answering many questions I got on several forums.

Title changed accordingly
« Last Edit: May 04, 2017, 08:23:42 AM by Nice Monkey »

Mount81

  • Senior Member
  • ***
  • Posts: 1098
    • View Profile
Hello!

I'm planning to buy a 3TB WD RED (WD30EFRX) in my Q10 Pro. Formatting to NTFS. The WD'S are made with AF, and as far as I have concluded right, the Q10 Pro can handle this combination...?!

Thanks for the answer!

Nice Monkey

  • Senior Member
  • ***
  • Posts: 1128
  • Loves classic movies
    • View Profile
Hello!

I'm planning to buy a 3TB WD RED (WD30EFRX) in my Q10 Pro. Formatting to NTFS. The WD'S are made with AF, and as far as I have concluded right, the Q10 Pro can handle this combination...?!

Thanks for the answer!
Should work just fine.

Using a Seagate Archive 8 TB myself due to best price/TB, which works fine too. Most (if not all) WD and Seagates drives can be used.
« Last Edit: July 27, 2017, 03:58:55 PM by Nice Monkey »

avenger89

  • Newbie
  • *
  • Posts: 25
    • View Profile
I recently got WD My Passport 4TB , Last time when i got i didnt know this and stored all my stuffs , only to find the himedia q5 doesnt read 4tb unless formatted properly.

After going through the post , i still dont know what to do.. sorry for that .. Can anyone tell me what i should format the new WD My Passport 4TB to ? NTFS ?

futeko.com

  • Administrator
  • Senior Member
  • *****
  • Posts: 4627
    • View Profile
GPT partition table and NTFS file system.

See here for a tutorial: http://www.futeko.com/newforum/index.php?topic=932.0

avenger89

  • Newbie
  • *
  • Posts: 25
    • View Profile
I checked both my old and new 4tb`s , both are GPT/NTFS formatted

https://i.imgur.com/vsOHTxx.png

Only thing is i dont remember formatting the old wd 4tb when it came as new. Should i have to freshly format again ?

osborn ronald

  • Newbie
  • *
  • Posts: 4
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or via USB enclosure)
« Reply #10 on: July 05, 2019, 10:00:00 AM »
Tutorial updated on several points. The main change is a negative advice for new HDD's with Native 4K Physical Sectors (4Kn) as these don't work SATA attached with the Hisillicon SOC/SDK used by Himedia and Egreat players. Maybe a driver update can solve this limitation but it can also be a HW limitation by the chipset itself. Older motherboards and drivers in PC's won't work with these 4Kn type HDD neither.


There are lots of different file systems. On Windows you are probably using NTFS, on OS X it is HFS+ and on Linux most likely ext4.
« Last Edit: November 25, 2019, 09:04:10 AM by futeko.com »

hjkaye

  • Newbie
  • *
  • Posts: 20
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or USB) and SDHC/SDXC Cards
« Reply #11 on: November 28, 2020, 11:15:49 PM »
Not sure if anyone is still monitoring this thread. I had been neglecting my Q10 pro for some time, I used it occasionally. Thought it was time to add some media and start using it again.

When I started again, I decided to upgrade the internal HDD from a 2TB to 4, when one went on sale near me. I chose a WD Blue 4tb WD40EZRZ. Per threads elsewhere I checked, this drive still has the 512b sectors.

The drive behaves VERY strangely. When I first installed and formatted, using the USB 3 connection on the back, it formatted fine using GPT and NTFS, was able to copy files no problem. I loaded up a bunch of ISO files, detached from the Q10 all was OK, I was able to see it in file manager, media center and VLC player.

I unmounted, re-attached USB, copied more files, ejected, detach USB, back to Q10, not detected!

I went through several iterations, formatting here and there with similar results.

I finally realized the sequence of events to failure is as follows:
  • Format the drive via back panel USB or external, and add as many files as I want
  • External/internal drive, doesn't matter, once installed in Q10 internal bay is OK
  • Unmounted and make ANY change, new folder, adding files doesn't seem to matter
  • Reconnect to internal, drive won't detect, settings/storage shows USB, corrupted.
  • still works fine via back panel USB on my Windows PC, chkdsk no errors
All would be ok if I never wanted to add or modify any files ever again.

The original 2TB drive operates fine. I did try a 2TB partition on the 4TB drive, I think all was ok, but I didn't take proper note of the results.

So, problem In a nutshell:

Via USB 3.0: Format/Add files/Eject - all OK
Unmounted, back to PC: Add/modify any files/folders whatsoever
Eject, detach USB cable/Return to internal - fails to mount
In settings/storage the drive shows as USB and corrupted.

Back to 2TB all ok... except running out of space. Very puzzling! Any ideas?

Nice Monkey

  • Senior Member
  • ***
  • Posts: 1128
  • Loves classic movies
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB up to 16 TB and SDHC/SDXC Cards
« Reply #12 on: February 25, 2022, 12:57:08 PM »
Updated the last chapter for HDD's > 16 TByte forming a new barrier with Android (not W10 64-bit).
My theory was confirmed with practical testing.

You are warned.
« Last Edit: March 02, 2022, 03:35:06 PM by Nice Monkey »

butch708

  • Newbie
  • *
  • Posts: 1
    • View Profile
hello Nice Monkey.

as you write in your post,
17,5 TByte = 16300 GB
i think you mean,
17,5 TB (Terabyte) = 16300 GiB (Gibibyte) or
163000 GB = 16,3 TB

I am not able to format my Seagate x18 18TB to 17,5 TB, max size is around 16,7 TB.

or am i understanding wrong?

And if i format my HDD in max size to ntfs, i get corupted filesystem from time to time in my zidoo z1000 pro.

ciao Steffen

turntable97

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or USB) and SDHC/SDXC Cards
« Reply #14 on: November 08, 2022, 04:08:51 PM »
Not sure if anyone is still monitoring this thread. I had been neglecting my Q10 pro for some time, I used it occasionally. Thought it was time to add some media and start using it again.

When I started again, I decided to upgrade the internal HDD from a 2TB to 4, when one went on sale near me. I chose a WD Blue 4tb WD40EZRZ. Per threads elsewhere I checked, this drive still has the 512b sectors.

The drive behaves VERY strangely. When I first installed and formatted, using the USB 3 connection on the back, it formatted fine using GPT and NTFS, was able to copy files no problem. I loaded up a bunch of ISO files, detached from the Q10 all was OK, I was able to see it in file manager, media center and VLC player.

I unmounted, re-attached USB, copied more files, ejected, detach USB, back to Q10, not detected!

I went through several iterations, formatting here and there with similar results.

I finally realized the sequence of events to failure is as follows:
  • Format the drive via back panel USB or external, and add as many files as I want
  • External/internal drive, doesn't matter, once installed in Q10 internal bay is OK
  • Unmounted and make ANY change, new folder, adding files doesn't seem to matter
  • Reconnect to internal, drive won't detect, settings/storage shows USB, corrupted.
  • still works fine via back panel USB on my Windows PC, chkdsk no errors
All would be ok if I never wanted to add or modify any files ever again.

The original 2TB drive operates fine. I did try a 2TB partition on the 4TB drive, I think all was ok, but I didn't take proper note of the results.

So, problem In a nutshell:

Via USB 3.0: Format/Add files/Eject - all OK
Unmounted, back to PC: Add/modify any files/folders whatsoever
Eject, detach USB cable/Return to internal - fails to mount
In settings/storage the drive shows as USB and corrupted.

Back to 2TB all ok... except running out of space. Very puzzling! Any ideas?

Hi,

I have basically the same problem.
I'm using a Samsung Extreme SSD with 1TB just fine. Never any issue.

But as I have recently upgraded to a Samsung Extreme Pro SSD with 4 TB, nothing works anymore.
I formatted the drive hundreds of times to NTFS or exFAT with different cluster sizes. Sometimes it's recognized, but as soon as I copy more files on the drive via Windows and reconnect it to my Dune HD RealVision 4K, it doesn't show up.

Quick short advice: Use a W10/W11 PC with the target HDD (E)SATA attached (may be a problem to find one):
- Be sure to remove all partions including any hidden partitions first
- Format a single partition with defaults using GPT + NTFS
Never found one that did not work next up to 16 TByte.

Part-1

OK finally figured out how that limit of > 2TB HDD support actually works and what can be done and what can not be done to overcome limitations.
Everything mentioned here was implemented by myself in practice. Practical experience includes using 2, 3, 4, 5, 6, 8, 10, 12 and 16 TB drives.

After reading this tutorial you will see that HDD > 2TB is supported both as Internal HDD and as USB attached HDD. There are however conditions to this and formats under both conditions in fact are (mostly) conflictive with each other.

I am afraid this is a bit technical, but as many of us including support staff was often confused about things, I will try to make it clear(er). If you are only interested in the do's and don'ts to make it work you may jump straight to Part-2.

As a start one needs to understand the difference between "HDD Initiation" and "HDD Formatting".
Often this is done in one step by a program, but the technical functions are different.
- The "HDD Initiation" tells the interface driver where and how to Boot, how the HDD can physically be accessed and where the physical partitions can be found on the disc. Here Microsoft Windows MBR or GPT initiations will be used to be compatibel with various Media Players
- "HDD Formatting" puts on the filesystem. This defines how and where files are put on the disk using as specific filesystem. Here the filesystem is assumed always to be Microsoft Windows NTFS to remain compatible with most Media Players, FAT32 works functionally but has a 4 GByte individual file size upper limit. That won't do for bigger MKV files and above all ISO images of DVD and BD. BD in fact will hardly ever work.

Some technical background for the interested
- First there are the Hard Drive manufacturers which make HDD drives traditonally read by sectors of 512 bytes each. With modern discs in Terabytes that is a lot of sectors which is not very efficient any more. Filesystems like FAT32 and NTFS therefore always read/write bigger blocks of data called clusters. A cluster is by default 4 KBytes (=8 sectors) and is configurable.

- Sectors are referenced to using 32-bit registers. If one calculates right one learns that a maximum 2.2 TByte can be addressed with a 32- bit counter adressing 512 byte sectors. Changing that is far from simple as this resides deep in the OS and Drivers. Partioning does not help here as still the whole disc space needs to be accessed at the physical level. This was up till recent done via the Master Boot Record (MBR) and Logical Block Addressing (LBA) using 32-bit. Windows changed to GPT (GUID Partion Table) with Windows-7 using 64-bit addressing solving this limitation structurally.

- But even with GPT the read/write overhead remains the same. So Hardware manufactures did a trick and intruduced Physical and Logical sectors. The newer 2 Tera Byte and bigger discs all work with Physical Sectors of 4K Bytes presented to the software as 8 Logical sectors of 512 bytes. This technology is called "Advanced Format 4K" which reduces the disc internal overhead but not the total CPU processing chain for I/O. The drives should be labeled 512e which stands for 512 byte logical sector emulation. To make a 512e HDD efficient sectors must be aligned at 4K boundaries. See the matching AF logo for a 512e HDD.

- The next step is Native 4K support. See attached logo for these 4Kn HDD. With 4Kn drives Logical Sectors, Physical Sectors and Clusters are all identical 4K Bytes streamlining throughput that way. This requires drivers dealing with 4K Logical sectors and obviously bolts on support of GPT for Initiation. Windows supports Native 4K HDD starting with Windows 8.1 OS. This paves the road for hassle-free support of > 2 TB and even > 16 TB HDD combined with improved performance. Starting begin 2016 the first Native 4K HDD's made by WD appeared on the market. These are sold as HDD or build-in into their USB3 Enclosures. Drives can easily be identified as both Logical Sectors and Physical Sectors are always 4K.

You can display all mentioned HDD format details using fsutil fsinfo ntfsinfo x: under Windows. This is a CMD to be executed with Admin rights! (Note x: replaces the actual drive being used)
You may also use "MiniTool Partition Wizard Free" (see bottom for more details).

Back to most media players around
Access can run via Internal HDD, USB HDD Docking Bay, USB HDD enclosure or a NAS.
- A NAS setup hides the total physical drive access plus format (RAID) used and as such supports > 2TB drives without the Media Player even being aware of it.

- Internal discs are SATA attached (like internal drives in your PC). Drivers for SATA are easier to adapt and most media players upgraded to GPT support. They can address > 2TB discs using Advanced Format 4K support. All external drives for mounting into a PC purchased in shops are formatted that way. There is always a buffering/memory limitation somewhere but there is not a hard limitation for the maximum disc size. We know that 6 TB and 8 TB drives do work, but I do expect drives far above 16 TB to work too.

- USB attached drives is a complete different ballgame. Drivers are in all places and often very old. Also 64-bit addressing support may be jeopardized by a structural HW limitation. So GPT support is often not yet available and worse not being planned for many products. Under the condition that GPT is handled by products on both ends then USB standards do not impose limitations. In practice only some recent USB3 Docking Bay implementations do include such support.
As a temporary fix hardware manufactures came up with a trick. They started to ship USB-enclosures (= Casing, USB/ATA bridge and a HDD as one unit) with native 4K Logical sector enabled. The USB/ATA bridge firmware presents these to the USB-side. Result MBR can now work with 32-bit adressing of 4K Logical sectors. The physical disk access is not a problem as this is done by the USB/ATA bridge here. This trick will obviously only work up to a limit of just above 16 TB capacity for older USB/ATA bridges but recent versions may support 64-bit LBA.

Part-2

Moving a HDD between Internal <=> External <=> NAS <=> Windows PC
Please use Microsoft Windows 10/11 only for re-formatting with GPT+NTFS and have it (E)SATA attached for Internal HDD's. I am sure that will work. MAC and Linux equivalents may do the trick, but never tried those myself. Nobody ever reported (good) results for media players
Internal HDD and USB HDD > 2TB are not inter-changeable and need to be re-initiated and re-formatted for each deployment. As most people don't open USB Drives/Housings/Enclosures they will never be aware of the difference. Personally I am now able to move any HDD in either direction without problems. This with the exception of (still exotic) 4Kn type HDD's having 4K Physical sectors not handled by Android players.

- A HDD pulled from a USB Drive/Housing/Enclosure will work only as an internal SATA attached drive again after standard re-initialisation + re-formatting. Obviously this HDD must be (E)SATA attached during mandatory re-initialisation + re-formatting using Windows-7 or equivalent standard tools. A common reason for doing so is that a complete USB-Drive may factually be cheaper than buying the same HDD as a separate unit. You still may be able to re-use the then empty USB-External enclosure later.

- A HDD pulled from a Plug & Play USB3 Docking Bay with GPT support (like Fantec's, Icy Box and probably Lindy's) will work straight as an Internal HDD again. In fact you may move individual units freely between such a Docking Bay and Internal bays of multiple brands. Tried this with sizes up to 8 TB. This won't work with RAID supporting variants obviously as these will need a re-build of many hours when a HDD is changed (even in JBOD mode).

- Formatting with 4K Logical sectors for USB External drives requires special tooling, it also helps if you exactly know what you are doing using such tools. I am using tooling from Acronis for this just as most USB HDD manufacturers do. You will run into this whenever you want to insert a new/re-used standard SATA internal HDD into an existing External USB Housing e.g. to increase its capacity.

- If you pull a HDD from a NAS then it won't work for either deployment as it has been re-formatted the wrong way. It may be using EXT3/EXT4 but in most cases will be using a proprietary format. Re-format it first for the purpose you have in mind being Internal or External USB enclosure as explained.

- A HDD pulled from a working Windows PC might still not work despite being formatted correctly with GPT + NTFS. The reason is Windows and/or the manufacturer has put an additional (hidden) partition on it for OS recovery purposes. Media players like to see a single Partition starting at the beginning of the physical disk. You need to remove all existing partitions first and next format again with GPT + NTFS when (E)SATA attached. Doing so via USB won't work in most cases neither as resulting HDD parameters are slightly different also not interpreted by Android.

Using (EXT2/EXT3) EXT4 formatted HDD in combination with MAC or Unix/Linux/Android based platforms.
First of all this is not officially supported by some media players but it is validated to be working just fine. This does not come as a real surprise as Android is a Linux derivation and Android kernels include support for EXT2/3/4.
EXT2 is often used on USB-sticks and SD-cards, but is less suitable (=less robust) for deployment on a HDD. Instead use EXT4 by preference.

Obviously EXT formatting has to deal with the same HDD technical developments as NTFS:
- EXT3 will support a HDD up to 16 TB using a Logical 4K block size The choice of the block size to be used may be automatic or a variable to be set manually; this depending on the OS, OS-version and the factual tool being used.
- EXT4 has no implicit HDD upper limit regarding total volume. Per OS and OS-version a support limit may be applicable and specified by the manufacturer. In practice EXT4 will work with any size HDD available within the foreseeable future.

SDHC/SDXC Cards and using an exFAT for HDD?
exFAT is the new defacto File System standard for SDXC cards. In fact exFAT was designed for memory cards and sticks and not for HDD. It is an excellent replacement for FAT32 removing all its practical limitations but not an advisable (=less robust) replacement for NTFS.
As SD-Cards are immensely popular this will move exFAT to almost any platform. You see already a lot of devices (above all phones) with SDXC/exFAT support included. Media Players are very slow to adopt SDXC card support. Using exFAT on SDHC cards should work in normal cases as it is a standard Android Lollipop feature by now.

Occasionally I read that only exFAT support makes the difference between SDHC and SDXC, this is a mistake as support for the new UHS-I bus is the real technical innovation coming with SDXC. This new SDXC bus has a far higher speed . The UHS-I backward compatible UHS-II bus will again double throughput and is almost as fast as SATA600 and has 2 rows of pins making identification easy. Some brands call everything up to 32 GByte SDHC and above 32 SDXC which is misleading also. I have seen 128 GByte SDHC cards not supporting the UHS-I bus. Look at the corresponding UHS-I/UHS-II speed labels which must be present for SDXC compatibility and performance indication. Media players will not work with native SDXC cards! Never seen any UHS-I bus enabled media players yet which is a pity.

SDHC/SDXC Hybrid Cards which also exist should always carry both a SDHC (e.g. @10) and SDXC (e.g. U1) speed indicator as they work when inserted in old SDHC slots and new SDXC slots, but in fact may not always carry both labels. SANdisk is known to make a real mess with labeling. See the attachment for a correctly labeled Kingston card. The term hybrid card which I use here is never used for any SD-card advertising which adds to the confusion.
When you e.g. put a 32 GByte Hybrid card (mostly labeled as SDHC) into a card-reader for SDXC with UHS support, then it will be blazing fast. The reverse is also true, putting a Hybrid card of 64+ GByte into an SDHC card-reader slot will slow it down to @10 speeds maximum even being an ultra fast UHS-II card. This will happen when using such a hybrid card with media players always.
When using a SDXC capable external USB Card Reader then it must be USB3 connected as transfer speeds are way above USB2 limits. Measured 85 MB/sec Read and 30 MB/sec Write on a 12 Euro 32 GB Hybrid card. This card works also with most of my media players. One player (Xtreamer eXpress) refuses to read hybrid cards for an unknown reason.

If you want to validate/view very detailed HDD settings yourself and/or to try various formats suitable for Media Players then you can use the excellent Windows based "MiniTool Partition Wizard Free".
https://www.partitionwizard.com/free-partition-manager.html
It allows HDD formatting with EXT2/EXT3/EXT4 using Windows too


Using Android based media players with disks > 16 TByte
Android uses a 32-bit LBA register to address Physical Sectors sectors causing the infamous HDD 2,2 TByte limit with 512 byte sectors. Using the 512e trick one can access just over 17,6 TByte with 4K Physical sectors emulating 8x 512 byte Logical Sectors hence 512e format.
Will be glad somebody convincing me being wrong here.

The unrelated cluster size is how big the OS writes the data single I/O in multiple logical sectors. Up to 32K allowed but may increase slack with many small files on it but also reducing I/O overhead with big/huge files like ISO's.

I have heard of persons being able to format 18 TByte and 20 TByte HDD's with NTFS and also claims with those drives to be 512e compatible but never working them on Android without problems next.

Using a 32-bit LBA counter with 4K Physical Sectors results in a 16 Tbyte GPT limit as far as I know. W10 64-bit and recent USB drives both support the essential 64-bit LBA adressing for new 18 ExaByte GPT total HDD limits. W10-64bit seems to allow 256 TB per partition with 128 partitions maximum not causing any practical limits for the foreseeable future.

Practical testing with Zidoo media players and a Seagate EXOS 18 TByte drive showed the limit to be just over 17.5 TByte (=16300 Gigabyte). Exactly as calculated by me. I do expect it to apply also to other drives over 16 TByte and all Android based media players.

You are warned!

@Nice Monkey
Thanks for your tutorial. But i just don't know what to do anymore. How can I get my 4TB USB-C SSD get to work on the Dune? There must be some way...

It would be great if someone has an idea and could help me. Thanks!  :)
« Last Edit: November 08, 2022, 04:26:18 PM by turntable97 »

 

DELIVERY ∙ PAYMENT ∙ WARRANTY+RETURNS ∙ ABOUT ∙ PRIVACY ∙ FORUM ∙ CONTACT
© SDHDIGITAL LTD 2024