[SOLVED] Extremely slow write speeds with AFP

[odedia]

Hi,

I am using Mac OS X Mountain Lion (GM).

My write speeds were VERY slow. An 11gb movie took over an hour to copy.

It took me a long time to figure out what is the root cause for my issue. I kept thinking it was the WD20EARS drives I'm using.

I have now switched from AFP to SMB, and the write speeds are excellent: less than 10 minutes for the same file.

There is something broken with the AFP protocol. It might be mountain lion, it might be Nas4free. Note that I switched to freenas 8 thinking the problem was only with the current release of nas4free, but the speed was the same.

Any help would be appriciated.



EDIT :

Ok, I seem to have found the issue.

It's the MTU setting. when I set MTU=9000 on nas4free and also set MTU=9000 in Mountain Lion, speeds jumped up really fast, it was as fast as copying to the internal drive. That's weird, since it wasn't necessary with freenas 0.7 and OS X Lion.

After setting this value, I had the very bad idea of trying to delete everything on the nas before beginning copying again

Why bad? Because deduplication and deleting a low of files without enough ram is VERY BAD FOR YOUR ZFS POOL:

http://mail.opensolaris.org/pipermail/z ... 50428.html

My server is now stuck on reboot due to:

pid 1025 (zpool), uid 0, was killed: out of swap space

I can't even access the nas from ssh anymore.

I will try to boot in safe mode when I get home and destroy the pool (not zfs itself, but zpool, see link above).

Then i'll try to recreate the pool without deduplication and see if that solves my problem.

can anyone recommend the optimal MTU setting for a realtek r8169 gigabit ethernet? Also, doesn't changing the value on Mountain Lion impact my internet connection which probably requires a 1500 value?

[sjordan]

That is very odd. I use nothing but AFP in my house and I can max out my 1Gig link for reads and writes (110MB/sec). My 3 macs using the AFP shares are Snow Leopard and Lion...I don't have any Mountain Lion installs yet (its still not officially released). Have you tried from any Snow Leopard or Lion installs?

[odedia]

I'm afraid I don't have a concrete way to test this, I have a lion machine in another room but that runs over wifi so not a good test.

I just tested another file and the speed is better, around 17 minutes for a 5.9gb file using gigabit ethernet.
Still, on the lion machine a similar sized file takes about the same time over wifi (with a slower machine).
On mountain lion, using NFS it takes only 6 minutes. There is clearly something wrong here.

I just noticed I posted this in the wrong forums, if anyone can move it to the AFP forum it'll be great.

[sjordan]

It should come close to matching the speeds of NFS or SMB....unless something has changed with Mountain Lion that is causing this. I am assuming your doing like for like test (same files and same machine). What build of Nas4Free are you running? I am on 141 and it has been flawless for me since day one.

EDIT: Also take a look at your log files and check for any afp errors

[odedia]

Nothing special the the logs.

I am running build 148.

OK, here's a test:

a 11.73gb file, being copied from direct-attached JBOD raid:

To internal Macbook Pro hard drive - around 5 minutes.
To NAS via NFS: around 17 minutes. top -P in shell shows that nfsd takes about 30% of cpu.
To NAS via AFP: "Time Left" indicates about 45 minutes. Testing a few minutes ago, it showed around 5 hours (!!) process afpd in the NAS shows a load of around 8%. in the test a few minutes ago, it showed a load of around 2%.

As I keep writing this, CPU load on the nas drops to around 6% while copy time increases to around 51 minutes. Can this be an Apple bug??

Screenshot:

http://dl.dropbox.com/u/45692/Screen%20 ... 0%20PM.png

Edit: this may not be relevant, but even the slighest change to a setting in the web gui resets the nfs access. I was simply changing the log lines order checkbox, clicked save, and it reset the hfs share.

[sjordan]

I really hope this is a fluke of your install and not an issue with Mountain Lion. I ordered a Retina MBP last week and plan to upgrade to Mountain Lion once it is released. If it were me I would take the Lion install you do have and plug it in via ethernet and see if you have the same issue. This will at least help you get closer to finding out where the fault is

[rocket]

I am running lion and just copied 56 gb in 11 minutes over AFP. I'm currently running build 139, however.

since Mountain Lion isn't actually released, you might expect to see changes.

[odedia]

Well, my machine was never the fastest - it is an atom processor with 4 gb of ram (3gb usable), but it was definitely much faster than what I have now.
I will test later and report back, still copying back all the data from external drives back to zfs. Only 1 more day for data copying (1tb via NDS) and 2 days for TIme Machine (via AFP).

Quick question: do you use dedup? I noticed that at these speeds it didn't change my load as much so I decided to keep it on, but I think dedup depends on the amount of data you have, right?

[sjordan]

I personally use it on one of my pools.....but I have a dedicated ssd for the l2arc (60Gig). I tried dedup with out the ssd and saw a big change in my transfer speeds. It depends on how much data and the type of data. Some people can get away with only need 1-2Gigs of dedicated ram/ssd for 1 TB while others will need more.

[odedia]

Ok, I seem to have found the issue.

It's the MTU setting. when I set MTU=9000 on nas4free and also set MTU=9000 in Mountain Lion, speeds jumped up really fast, it was as fast as copying to the internal drive. That's weird, since it wasn't necessary with freenas 0.7 and OS X Lion.

After setting this value, I had the very bad idea of trying to delete everything on the nas before beginning copying again

Why bad? Because deduplication and deleting a low of files without enough ram is VERY BAD FOR YOUR ZFS POOL:

http://mail.opensolaris.org/pipermail/z ... 50428.html

My server is now stuck on reboot due to:

pid 1025 (zpool), uid 0, was killed: out of swap space

I can't even access the nas from ssh anymore.

I will try to boot in safe mode when I get home and destroy the pool (not zfs itself, but zpool, see link above).

Then i'll try to recreate the pool without deduplication and see if that solves my problem.

can anyone recommend the optimal MTU setting for a realtek r8169 gigabit ethernet? Also, doesn't changing the value on Mountain Lion impact my internet connection which probably requires a 1500 value?

[odedia]

Ok, changes applied. I had to literally overwrite my GPT header data on the hard drive to be able to reuse these drives.

I am now happily copying over 2.2tb of data to the NAS. It will take less than 20 hours. What a HUGE difference that made!

I would appreciate comments from people who know regarding my internet access question. Most websites seem to work just fine with MTU set to 9000, but for some reason facebook.com refuses to open. I wonder if forcing nas4free to 1500 MTU would work just like 9000 MTU. But I don't plan on testing anything until my data is back on the drives.

[sjordan]

From what I understand the setting is dependent on the NIC's (what they can handle) and your switch. I have my OSX machines set to 9000, and my nas4free set to 9000....I also have a switch that is capable of jumbo frames. Not that this is important, but I can hit facebook.com just fine from my OSX Machines.

Look up your switch and see if it is capable of 9000. If it is not then you will have to set your MTU across the board to match what your switch can handle.

[odedia]

My Laptop is a 2011 macbook pro that can handle 9000, my nas has an r8169 realtek that can handle large frames (not sure if 9000 is optimal but so far looks great), and my switch is setup a little weird due to the way my country's ISP system works:

I have an Apple Time Capsule with gigabit ethernet switch that is the main switch in my house, the nas and laptop are connected to that.

To the WAN port on the Time Capsule, i'm connecting the ISP's router (it is a fixed modem+router box), so all the internet traffic goes through that router and is being routed to the time capsule's WAN port. The ISP's box does not support jumbo frames, I'm pretty sure.

[odedia]

The last piece of the puzzle has now been solved - the issue was with OpenDNS which I've been using for the last several months.

I switched back to my ISP's DNS address and all sites are accessible again. The internet actually feels a lot faster!

Thanks to everyone for all the help. This was quite an educational week for me on ZFS, NAS4FREE and networking!