UK BASED
FREE WORLD SHIPPING
2 YEAR WARRANTY
FEATURED  :  MEDIA BOX  :  BUNDLES  :  OTHER  :  FORUM  :  CART
Zidoo Z10
3.5" HDD BAY
REFRESH+RESOLUTION SWITCHING
UHD+BD+DVD ISO FULL MENUS
ZIDOO HOME THEATRE
IN STOCK

Author Topic: Tutorial: Using HDD with sizes > 2 TB (Internal or USB) and SDHC/SDXC Cards  (Read 8215 times)

Nice Monkey

  • Senior Member
  • ***
  • Posts: 695
  • Loves classic movies
    • View Profile
Part-1

OK finally figured out how that limit of > 2TB HDD support actually works and what can be done and what can not be done to overcome limitations.
Everything mentioned here was implemented by myself in practice. Practical experience includes using 2, 3, 4, 5, 6 and 8 TB drives.

After reading this tutorial you will see that HDD > 2TB is supported both as Internal HDD and as USB attached HDD. There are however conditions to this and formats under both conditions in fact are (mostly) conflictive with each other.

I am afraid this is a bit technical, but as many of us including support staff was often confused about things, I will try to make it clear(er). If you are only interested in the do's and don'ts to make it work you may jump straight to Part-2.

As a start one needs to understand the difference between "HDD Initiation" and "HDD Formatting".
Often this is done in one step by a program, but the technical functions are different.
- The "HDD Initiation" tells the interface driver where and how to Boot, how the HDD can physically be addressed and where the physical partitions can be found on the disc. Here Microsoft Windows MBR or GPT initiations will be used to be compatibel with various Media Players
- "HDD Formatting" puts on the filesystem. This defines how and where files are put on the disk using as specific filesystem. Here the filesystem is assumed always to be Microsoft Windows NTFS to remain compatible with most Media Players, FAT32 works functionally but has a 4 GByte individual file size upper limit. That won't do for bigger MKV files and above all ISO images of DVD and BD. BD in fact will hardly ever work.

Some technical background for the interested
- First there are the Hard Drive manufacturers which make HDD drives traditonally read by sectors of 512 bytes each. With modern discs in Terabytes that is a lot of sectors which is not very efficient any more. Filesystems like FAT32 and NTFS therefore always read/write bigger blocks of data called clusters. A cluster is by default 4 KBytes (=8 sectors) and is configurable.

- Sectors are referenced to using 32-bit registers. If one calculates right one learns that a maximum 2.2 TByte can be addressed with a 32- bit counter adressing 512 byte sectors. Changing that is far from simple as this resides deep in the OS and Drivers. Partioning does not help here as still the whole disc space needs to be accessed at the physical level. This was up till recent done via the Master Boot Record (MBR) and Logical Block Addressing (LBA) using 32-bit. Windows changed to GPT (GUID Partion Table) with Windows-7 using 64-bit addressing solving this limitation structurally.

- But even with GPT the read/write overhead remains the same. So Hardware manufactures did a trick and intruduced Physical and Logical sectors. The newer 2 Tera Byte and bigger discs all work with Physical Sectors of 4K Bytes presented to the software as 8 Logical sectors of 512 bytes. This technology is called "Advanced Format 4K" which reduces the disc internal overhead but not the total CPU processing chain for I/O. The drives should be labeled 512e which stands for 512 byte logical sector emulation. To make a 512e HDD efficient sectors must be aligned at 4K boundaries. See the matching AF logo for a 512e HDD.

- The next step is Native 4K support. See attached logo for these 4Kn HDD. With 4Kn drives Logical Sectors, Physical Sectors and Clusters are all identical 4K Bytes streamlining throughput that way. This requires drivers dealing with 4K Logical sectors and obviously bolts on support of GPT for Initiation. Windows supports Native 4K HDD starting with Windows 8.1 OS. This paves the road for hazzlefree support of > 2 TB and even > 16 TB HDD combined with improved performance. Starting begin 2016 the first Native 4K HDD's made by WD appeared on the market. These are sold as HDD or build-in into their USB3 Enclosures. Drives can easily be identified as both Logical Sectors and Physical Sectors are always 4K.

You can display all mentioned HDD format details using fsutil fsinfo ntfsinfo x: under Windows. This is a CMD to be executed with Admin rights! (Note x: replaces the actual drive being used)
You may also use "MiniTool Partition Wizard Free" (see bottom for more details).

Back to most media players around
Access can run via Internal HDD, USB HDD Docking Bay, USB HDD enclosure or a NAS.
- A NAS setup hides the total physical drive access plus format (RAID) used and as such supports > 2TB drives without the Media Player even being aware of it.

- Internal discs are SATA attached (like internal drives in your PC). Drivers for SATA are easier to adapt and most media players upgraded to GPT support. They can address > 2TB discs using Advanced Format 4K support. All external drives for mounting into a PC purchased in shops are formatted that way. There is always a buffering/memory limitation somewhere but there is not a hard limitation for the maximum disc size. We know that 6 TB and 8 TB drives do work, but I do expect drives far above 16 TB to work too.

- USB attached drives is a complete different ballgame. Drivers are in all places and often very old. Also 64-bit addressing support may be jeopardized by a structural HW limitation. So GPT support is often not yet available and worse not being planned for many products. Under the condition that GPT is handled by products on both ends then USB standards do not impose limitations. In practice only some recent USB3 Docking Bay implementations do include such support.
As a temporary fix hardware manufactures came up with a trick. They started to ship USB-enclosures (= Casing, USB/ATA bridge and a HDD as one unit) with native 4K Logical sector enabled. The USB/ATA bridge firmware presents these to the USB-side. Result MBR can now work with 32-bit adressing of 4K Logical sectors. The physical disk access is not a problem as this is done by the USB/ATA bridge here. This trick will obviously only work up to a limit of just above 16 TB capacity.

Part-2

Moving a HDD between Internal <=> External <=> NAS <=> Windows PC
Please use Microsoft Windows 7/8/10 only for re-formatting with GPT+NTFS and have it (E)SATA attached for Internal HDD's. I am sure that will work. MAC and Linux equivalents may do the trick, but never tried those myself. Nobody ever reported (good) results for media players
Internal HDD and USB HDD > 2TB are not inter-changeable and need to be re-initiated and re-formatted for each deployment. As most people don't open USB Drives/Housings/Enclosures they will never be aware of the difference. Personally I am now able to move any HDD in either direction without problems. This with the exception of (still exotic) 4Kn type HDD's having 4K Physical sectors not handled by Android players.

- A HDD pulled from a USB Drive/Housing/Enclosure will work only as an internal SATA attached drive again after standard re-initialisation + re-formatting. Obviously this HDD must be (E)SATA attached during mandatory re-initialisation + re-formatting using Windows-7 or equivalent standard tools. A common reason for doing so is that a complete USB-Drive may factually be cheaper than buying the same HDD as a separate unit. You still may be able to re-use the then empty USB-External enclosure later.

- A HDD pulled from a Plug & Play USB3 Docking Bay with GPT support (like Fantec's, Icy Box and probably Lindy's) will work straight as an Internal HDD again. In fact you may move individual units freely between such a Docking Bay and Internal bays of multiple brands. Tried this with sizes up to 8 TB. This won't work with RAID supporting variants obviously as these will need a re-build of many hours when a HDD is changed (even in JBOD mode).

- Formatting with 4K Logical sectors for USB External drives requires special tooling, it also helps if you exactly know what you are doing using such tools. I am using tooling from Acronis for this just as most USB HDD manufacturers do. You will run into this whenever you want to insert a new/re-used standard SATA internal HDD into an existing External USB Housing e.g. to increase its capacity.

- If you pull a HDD from a NAS then it won't work for either deployment as it has been re-formatted the wrong way. It may be using EXT3/EXT4 but in most cases will be using a proprietary format. Re-format it first for the purpose you have in mind being Internal or External USB enclosure as explained.

- A HDD pulled from a working Windows PC might still not work despite being formatted correctly with GPT + NTFS. The reason is Windows and/or the manufacturer has put an additional (hidden) partition on it for OS recovery purposes. Media players like to see a single Partition starting at the beginning of the physical disk. You need to remove all existing partitions first and next format again with GPT + NTFS when (E)SATA attached. Doing so via USB won't work in most cases neither as resulting HDD parameters are slightly different also not interpreted by Android.

Using (EXT2/EXT3) EXT4 formatted HDD in combination with MAC or Unix/Linux/Android based platforms.
First of all this is not officially supported by some media players but it is validated to be working just fine. This does not come as a real surprise as Android is a Linux derivation and Android kernels include support for EXT2/3/4.
EXT2 is often used on USB-sticks and SD-cards, but is less suitable (=less robust) for deployment on a HDD use EXT4 by preference.

Obviously EXT formatting has to deal with the same HDD technical developments as NTFS:
- EXT3 will support a HDD up to 16 TB using a Logical 4K block size The choice of the block size to be used may be automatic or a variable to be set manually; this depending on the OS, OS-version and the factual tool being used.
- EXT4 has no implicit HDD upper limit regarding total volume. Per OS and OS-version a support limit may be applicable and specified by the manufacturer. In practice EXT4 will work with any size HDD available within the foreseeable future.

SDHC/SDXC Cards and using an exFAT for HDD?
exFAT is the new defacto File System standard for SDXC cards. In fact exFAT was designed for memory cards and sticks and not for HDD. It is an excellent replacement for FAT32 removing all its practical limitations but not an advisable (=less robust) replacement for NTFS.
As SD-Cards are immensely popular this will move exFAT to almost any platform. You see already a lot of devices (above all phones) with SDXC/exFAT support included. Media Players are very slow to adopt SDXC card support. Using exFAT on SDHC cards should work in normal cases as it is a standard Android Lollipop feature by now.

Occasionally I read that only exFAT support makes the difference between SDHC and SDXC, this is a mistake as support for the new UHS-I bus is the real technical innovation coming with SDXC. This new SDXC bus has a far higher speed . The UHS-I backward compatible UHS-II bus will again double throughput and is almost as fast as SATA600 and has 2 rows of pins making identification easy. Some brands call everything up to 32 GByte SDHC and above 32 SDXC which is misleading also. I have seen 128 GByte SDHC cards not supporting the UHS-I bus. Look at the corresponding UHS-I/UHS-II speed labels which must be present for SDXC compatibility and performance indication. Media players will not work with native SDXC cards! Never seen any UHS-I bus enabled media players yet which is a pity.

SDHC/SDXC Hybrid Cards which also exist should always carry both a SDHC (e.g. @10) and SDXC (e.g. U1) speed indicator as they work when inserted in old SDHC slots and new SDXC slots, but in fact may not always carry both labels. SANdisk is known to make a real mess with labeling. See the attachment for a correctly labeled Kingston card. The term hybrid card which I use here is never used for any SD-card advertising which adds to the confusion.
When you e.g. put a 32 GByte Hybrid card (mostly labeled as SDHC) into a card-reader for SDXC with UHS support, then it will be blazing fast. The reverse is also true, putting a Hybrid card of 64+ GByte into an SDHC card-reader slot will slow it down to @10 speeds maximum even being an ultra fast UHS-II card. This will happen when using such a hybrid card with media players always.
When using a SDXC capable external USB Card Reader then it must be USB3 connected as transfer speeds are way above USB2 limits. Measured 85 MB/sec Read and 30 MB/sec Write on a 12 Euro 32 GB Hybrid card. This card works also with most of my media players. One player (Xtreamer eXpress) refuses to read hybrid cards for an unknown reason.

If you want to validate/view very detailed HDD settings yourself and/or to try various formats suitable for Media Players then you can use the excellent Windows based "MiniTool Partition Wizard Free 10.2".
https://www.partitionwizard.com/free-partition-manager.html
It allows HDD formatting with EXT2/EXT3/EXT4 using Windows too
« Last Edit: May 04, 2017, 08:23:02 AM by Nice Monkey »

futeko.com

  • Administrator
  • Senior Member
  • *****
  • Posts: 3959
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or via USB enclosure)
« Reply #1 on: February 04, 2016, 02:56:04 PM »
Excellent tutorial. Thanks.

Nice Monkey

  • Senior Member
  • ***
  • Posts: 695
  • Loves classic movies
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or via USB enclosure)
« Reply #2 on: February 22, 2016, 08:08:23 AM »
Warning: Western Digital recently also ships both  1 and 2 TB Internal and USB External HDD enclosures with 512e. I am a bit puzzled why they decided to do so. Could be for performance reasons or  just to simplify (standardize) their production? Anyway it will confuse owners of media players even more. On PC' s the problem is virtually gone as all old OS having problems are obsolete now and most Windows OS have been upgraded to W10 by now.

Toshiba is shipping Native 4K drives (4Kn). Did not get any of these to work with Egreat A10 players SATA. As these use the same SOC as Himedia I doubt they will work with Q10 Pro. Dealing with the Physical 4K sectors seem to cause the incompatibility. They do work via USB though.

Use the display methods documented to reveal if you are running into this problem.
« Last Edit: March 29, 2017, 07:39:14 AM by Nice Monkey »

Nice Monkey

  • Senior Member
  • ***
  • Posts: 695
  • Loves classic movies
    • View Profile
Re: Tutorial: Using HDD with sizes > 2 TB (Internal or via USB enclosure)
« Reply #3 on: March 29, 2017, 08:00:04 AM »
Tutorial updated on several points. The main change is a negative advice for new HDD's with Native 4K Physical Sectors (4Kn) as these don't work SATA attached with the Hisillicon SOC/SDK used by Himedia and Egreat players. Maybe a driver update can solve this limitation but it can also be a HW limitation by the chipset itself. Older motherboards and drivers in PC's won't work with these 4Kn type HDD neither.
« Last Edit: March 29, 2017, 08:03:40 AM by Nice Monkey »

Nice Monkey

  • Senior Member
  • ***
  • Posts: 695
  • Loves classic movies
    • View Profile
Updated the post with SDHC/SDXC and Hybrid SD-card support now that people start to use these also for Internal Android storage expansion.
Overall  textual update answering many questions I got on several forums.

Title changed accordingly
« Last Edit: May 04, 2017, 08:23:42 AM by Nice Monkey »

Mount81

  • Senior Member
  • ***
  • Posts: 803
    • View Profile
Hello!

I'm planning to buy a 3TB WD RED (WD30EFRX) in my Q10 Pro. Formatting to NTFS. The WD'S are made with AF, and as far as I have concluded right, the Q10 Pro can handle this combination...?!

Thanks for the answer!

Nice Monkey

  • Senior Member
  • ***
  • Posts: 695
  • Loves classic movies
    • View Profile
Hello!

I'm planning to buy a 3TB WD RED (WD30EFRX) in my Q10 Pro. Formatting to NTFS. The WD'S are made with AF, and as far as I have concluded right, the Q10 Pro can handle this combination...?!

Thanks for the answer!
Should work just fine.

Using a Seagate Archive 8 TB myself due to best price/TB, which works fine too. Most (if not all) WD and Seagates drives can be used.
« Last Edit: July 27, 2017, 03:58:55 PM by Nice Monkey »

avenger89

  • Newbie
  • *
  • Posts: 7
    • View Profile
I recently got WD My Passport 4TB , Last time when i got i didnt know this and stored all my stuffs , only to find the himedia q5 doesnt read 4tb unless formatted properly.

After going through the post , i still dont know what to do.. sorry for that .. Can anyone tell me what i should format the new WD My Passport 4TB to ? NTFS ?

futeko.com

  • Administrator
  • Senior Member
  • *****
  • Posts: 3959
    • View Profile
GPT partition table and NTFS file system.

See here for a tutorial: http://www.futeko.com/newforum/index.php?topic=932.0

avenger89

  • Newbie
  • *
  • Posts: 7
    • View Profile
I checked both my old and new 4tb`s , both are GPT/NTFS formatted

https://i.imgur.com/vsOHTxx.png

Only thing is i dont remember formatting the old wd 4tb when it came as new. Should i have to freshly format again ?

 

Zidoo X20 Pro
PRO DAC
2X HDMI
2X 3.5" HDD BAY
REFRESH+RESOLUTION SWITCHING
UHD+BD+DVD ISO FULL MENUS
DELIVERY  :  WARRANTY  :  RETURNS  :  FAQ  :  BREXIT  :  ABOUT  :  CONTACT
PayPal Acceptance Mark Prices are charged in £GBP with conversion at Paypal's current rate. The Euro / $USD price on our site is approximate. More info here. Prices for UK + EU delivery include VAT at 20%. COOKIE POLICY.

© SDHDIGITAL LTD 2018. All rights reserved.