Comments on: PHP Libraries Hate Ram http://www.symphonious.net/2009/05/25/php-libraries-hate-ram/ Living in a state of accord. Tue, 07 Feb 2012 01:07:58 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Adrian Suttonhttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174074 Adrian Sutton Tue, 26 May 2009 08:38:39 +0000 http://www.symphonious.net/?p=1179#comment-174074 Yeah, I've read the docs on the stream support, it's pretty good, but as you mentioned a lot of stuff doesn't use it which makes it far less useful. The Java stream stuff is long winded (moreso than the php stuff) but because it's been there from day one and most of the Java libraries were developed purely in Java rather than using existing C libraries, it's very comprehensively used. Such is life. Yeah, I’ve read the docs on the stream support, it’s pretty good, but as you mentioned a lot of stuff doesn’t use it which makes it far less useful. The Java stream stuff is long winded (moreso than the php stuff) but because it’s been there from day one and most of the Java libraries were developed purely in Java rather than using existing C libraries, it’s very comprehensively used.

Such is life.

]]> By: Pierrehttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174073 Pierre Tue, 26 May 2009 08:26:40 +0000 http://www.symphonious.net/?p=1179#comment-174073 Right Jon, that was my thoughts too. About php having no clue about stream, I think you should really consider to actually read the docs about the stream API in PHP (userland or internals). It is one of the most powerful out there. The problem is that some extensions do not rely on it (due to the underlying libraries for example), but the stream API itself in PHP is great, try it. Right Jon, that was my thoughts too.

About php having no clue about stream, I think you should really consider to actually read the docs about the stream API in PHP (userland or internals). It is one of the most powerful out there. The problem is that some extensions do not rely on it (due to the underlying libraries for example), but the stream API itself in PHP is great, try it.

]]> By: Adrian Suttonhttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174072 Adrian Sutton Mon, 25 May 2009 20:40:39 +0000 http://www.symphonious.net/?p=1179#comment-174072 Ah good point Jon. I hadn't thought about the lack of a known content length for the browser's progress bar. That makes me feel better about having to use temporary files. The zip files that are created are just thrown out once the download finishes in this case, since they're fairly highly customised. Depending on traffic/load we can always investigate a caching layer to keep them around for longer. Ah good point Jon. I hadn’t thought about the lack of a known content length for the browser’s progress bar. That makes me feel better about having to use temporary files. The zip files that are created are just thrown out once the download finishes in this case, since they’re fairly highly customised. Depending on traffic/load we can always investigate a caching layer to keep them around for longer.

]]> By: Jonhttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174071 Jon Mon, 25 May 2009 20:11:17 +0000 http://www.symphonious.net/?p=1179#comment-174071 I have a lot of experience with this problem because the company I work for (Kink.com) offers .zip file downloads of all of our content. The issue is that it nearly impossible (ie: not reliable) to create a reliable streaming option for zip files because it is nearly impossible to know the final length of the file before the entire file has been created. The zip specification is crappy in this regard. The reason you need this information is that you need to be able to set the Content-Length: header so that downloads don't look like they are unlimited in the browser download window. People like to have an idea of when a download will complete. I tried implementing a dynamic streaming solution in PHP, Python and Java. PHP and Python have no clue about streaming (which you figured out already). They love to hold data in memory which is terrible for trying to create a large scale concurrent streaming system. Java does streaming really well, but it is impossible to know the total length of the resulting file until after it is created. On top of it, each file tends to get created with random sizes (even without compression), so you can't cache this data anywhere. We ended up punting on all of this and just create and cache the entire zip files. They get copied out to our CDN with the rest of our video/picture data. It doubled our storage requirements, but what's a few extra TB these days. =) I have a lot of experience with this problem because the company I work for (Kink.com) offers .zip file downloads of all of our content. The issue is that it nearly impossible (ie: not reliable) to create a reliable streaming option for zip files because it is nearly impossible to know the final length of the file before the entire file has been created. The zip specification is crappy in this regard. The reason you need this information is that you need to be able to set the Content-Length: header so that downloads don’t look like they are unlimited in the browser download window. People like to have an idea of when a download will complete.

I tried implementing a dynamic streaming solution in PHP, Python and Java. PHP and Python have no clue about streaming (which you figured out already). They love to hold data in memory which is terrible for trying to create a large scale concurrent streaming system. Java does streaming really well, but it is impossible to know the total length of the resulting file until after it is created. On top of it, each file tends to get created with random sizes (even without compression), so you can’t cache this data anywhere.

We ended up punting on all of this and just create and cache the entire zip files. They get copied out to our CDN with the rest of our video/picture data. It doubled our storage requirements, but what’s a few extra TB these days. =)

]]> By: Adrian Suttonhttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174068 Adrian Sutton Mon, 25 May 2009 12:32:15 +0000 http://www.symphonious.net/?p=1179#comment-174068 True, I don't know why I was thinking PHP was GPL.... True, I don’t know why I was thinking PHP was GPL….

]]> By: Pierrehttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174067 Pierre Mon, 25 May 2009 12:21:34 +0000 http://www.symphonious.net/?p=1179#comment-174067 oh, I won't read the sources but see how it works from a runtime pov. PHP license is not GPL compatible so I won't taint my code with some possible GPL code (lawers--) :) oh, I won’t read the sources but see how it works from a runtime pov. PHP license is not GPL compatible so I won’t taint my code with some possible GPL code (lawers–) :)

]]> By: Adrian Suttonhttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174066 Adrian Sutton Mon, 25 May 2009 12:19:48 +0000 http://www.symphonious.net/?p=1179#comment-174066 Cool. The OpenJDK project provides the Java source code under GPL license so that's probably your best bet. It's also included with the JDK download, but under a proprietary license which at least in the past had some pretty overbearing clauses about tainting etc. Not sure if that got cleaned up at some point, but OpenJDK is a safe bet anyway. Cool. The OpenJDK project provides the Java source code under GPL license so that’s probably your best bet. It’s also included with the JDK download, but under a proprietary license which at least in the past had some pretty overbearing clauses about tainting etc. Not sure if that got cleaned up at some point, but OpenJDK is a safe bet anyway.

]]> By: Pierrehttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174065 Pierre Mon, 25 May 2009 12:05:13 +0000 http://www.symphonious.net/?p=1179#comment-174065 Interesting, I'd to take a look and see if I can add this kind of features to ext/zip (I maintain it). Thanks for the discussions&suggestions! Interesting, I’d to take a look and see if I can add this kind of features to ext/zip (I maintain it). Thanks for the discussions&suggestions!

]]> By: Adrian Suttonhttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174064 Adrian Sutton Mon, 25 May 2009 11:12:13 +0000 http://www.symphonious.net/?p=1179#comment-174064 I'm sure it doesn't create a temporary file. When closing off the zip index you don't need the file content - just some of the meta data (name, size, date etc). It's pretty simple to store just that meta data in memory instead of the entire file so you can write the index at the end. I’m sure it doesn’t create a temporary file. When closing off the zip index you don’t need the file content – just some of the meta data (name, size, date etc). It’s pretty simple to store just that meta data in memory instead of the entire file so you can write the index at the end.

]]> By: Pierrehttp://www.symphonious.net/2009/05/25/php-libraries-hate-ram/comment-page-1/#comment-174063 Pierre Mon, 25 May 2009 09:52:15 +0000 http://www.symphonious.net/?p=1179#comment-174063 Yes, but are you sure that it does not create a temporary file? It sounds very hard (impossible) to finalize an archive index when the archive content is not even known. Yes, but are you sure that it does not create a temporary file?

It sounds very hard (impossible) to finalize an archive index when the archive content is not even known.

]]>