Age: 2398.36d Health: 100% Posters: 33 Posts: 137 Replies: 109 Files: 0
|>>||!///SWFAnts #ADMIN# 25mar2012(su)10:00 No.3368 SWF |P1
|Let's talk Services|
All discussion related to the services found at http://eye.swfchan.com/services/ are to be put in this thread.
Comments, questions, suggestions, ideas, wishes, problems?
|>>||Anonymous 26mar2012(mo)06:44 No.3382 A |P2R1
|I'd love to see a video tutorial on how to properly make use of the all the cool services here on this board; do you think a quick "Swfchan 101" video would be appropriate?|
|>>||!///SWFAnts #ADMIN# 27mar2012(tu)07:49 No.3383 SWF |P3R2
I was thinking about it the other day, and now that I know people actually want it I'll think even more about it. Although a bit weird it might be a very good idea to have some sort of intro video to the site at this point. swfchan is losing new visitors to simpler main-stream sites more and more every day, maybe having a video would help to catch some new blood. If anybody would watch it that is, haha.
|>>||Anonymous 27mar2012(tu)09:06 No.3384 A |P4R3
|I certainly would. I love this board's features so much but I wish I could take full advantage of them.|
SWFCHAN 4 LIFE. :D
|>>||!///SWFAnts #ADMIN# 27mar2012(tu)12:38 No.3386 SWF |P5R4
Is there any feature in particular I can help you understand better? If you have a question about anything at all feel free to ask. :)
|>>||Anonymous 28mar2012(we)00:36 No.3388 A |P6R5
|Just a general overview would be nice, like I said, I don't really know how to use a lot of the features on this board... :P|
Thanks, by the way.
|>>||Maharene 27apr2012(fr)18:19 No.3531 B |P7R6
Morning! I had shut down the mirror while I was at school, loading it back up today and I'm having the same issue others were having yesterday. It's been approximately 10 minutes since receiving:
27apr2012 12:02:18 Listening on 12345 (UDP) and 54321 (TCP).
27apr2012 12:02:20 Server: Your mirror is up and running. Thanks!
And I have not received the reflecting message. I've tried to reboot the .jar file a couple times and haven't been able to reflect the flashes. I've checked my router and the ports are still open so I'm not sure why today is different from yesterday. Any idea what this could be?
|>>||Maharene 28apr2012(sa)00:46 No.3532 B |P8
|27apr2012 18:44:25 Server: Your mirror is up and running. Thanks!|
27apr2012 18:44:40 Reflecting 47 flashes on swfchan.
Suddenly success? Have been restarting every hour or so, finally some response.
|>>||Anonny 28apr2012(sa)23:55 No.3533 C |P9R7
|Yea, it doesn't do it right away for me. I usually just let it sit there while I do other stuff and, eventually, it goes up.|
|>>||!///SWFAnts #ADMIN# 29apr2012(su)04:14 No.3535 SWF |P10R8
|The "thanks" message is displayed when the client and swfchan has been able to communicate both ways via UDP. The "reflecting" message is displayed after swfchan has downloaded a list of shared hashes via TCP from the client, and after the mirror database on the server has been successfully updated with the knowledge of what flashes you are sharing.|
If the server can't download a client's list of shared hashes it should notify it via UDP (something like "check your TCP settings"). If you guys are getting the "thanks" message but NOT the "check your TCP settings" it means that something must be wrong between receiving a client's hashes and inserting them into the database. Either that or the server does not try to get the list of hashes, possibly due to a cached instance of you in its memory (it may think it already have your hashes).
Will look into it as soon as I have time to sit down and code on this again. Maybe I will implement an auto-update feature in the future sometime so you guys won't have to perform the update manually. Depends on if it's worth it, eventually I'll hit a point when the program won't need to be updated that much, if at all. I've already got a somewhat long TODO list on the mirror project.
I appreciate that you guys are running the clients! Do you get a lot of traffic? How many uploads did you do in the last 24 hours for example? Not counting time after when you didn't get a "reflecting" message.
|>>||Anonny 29apr2012(su)22:24 No.3537 C |P11R9
|I've probably done over 100 uploads since the time I first got it, but that's just guessing, it may be more.|
|>>||Brony 30apr2012(mo)05:47 No.3539 D |P12R10
Okay, I've downloaded the .rar file but I have no idea how to move files from swfchan to the .rar file to help with the mirroring. Once I can get it running I'm more than happy to host as many flashes as I possibly can. Two things, though.... Can the .rar file share my personal info? And also, what could somebody do with my IP address?
|>>||Anonny 30apr2012(mo)08:52 No.3540 C |P13R11
>Can the .rar file share my personal info?
I do believe Mackan can show you through the steps of doing this.
>And also, what could somebody do with my IP address?
Unless you have an unsecure server on your computer (which you probably do not as this is not an unsecure server), or you are hosting you're own internet and do not have an ISP (also highly, highly doubtful), you are completely in the clear. You can't do jack shit with IP addresses, contrary to what the CSI and news shows tell you.
|>>||Anonny 30apr2012(mo)08:53 No.3541 C |P14
>Can the .rar file share my personal info?
>I do believe Mackan can show you through the steps of doing this.
I am deeply sorry for the misreading, I was referencing your question on how to set the server up from a .rar file. As to your actual question, no. A rar file is just like a folder except it's compressed into an actual file.
|>>||!///SWFAnts #ADMIN# 1may2012(tu)01:30 No.3545 SWF |P15R12
Great that you want to put up a mirror! I added a how-to guide on http://mirror.swfchan.com/, please have a look at it.
The RAR file is just a compressed file that contains the JAR file. The JAR file is the actual program. Perhaps I should just put up the JAR directly, compressing it is really just complicating things. The only reason to put it in a RAR file is to make sure the extracted file is put into its own folder, or else it may become mixed with a lot of other downloaded files (not a big deal but it can become ugly since the JAR file creates a couple of files/folders of its own, it's just better to have it in its own folder). Also there might be more files added to the client than just the JAR in the future (although I can't think of any right now). For now I'll stick with the archive.
Anyway, no personal info is shared about you. There's no code in the JAR that can gather such info about you in the first place. All the program knows about is your share and your settings.
As for what people can do with your IP, it's not much. As long as your OS has the latest security updates and your firewall is working you should be safe from wannabe hackers. And if someone "tracks" your IP all they will get is the location of your Internet Service Provider, which already is pretty public to anyone.
|>>||Maharene 1may2012(tu)04:25 No.3547 B |P16R13
I don't have an estimate daily for the mirror quite yet, simply because I don't have it up unless I'm home. I can get that data for you on/after Thursday when I am home all day to monitor it.
I really came on to say that the new version is much more secure! I haven't had an aborted download since applying it. Thank you!
|>>||!///SWFAnts #ADMIN# 1may2012(tu)22:15 No.3548 SWF |P17R14
That's great! And no need to thank me, instead I'll thank you for having a mirror up. And everybody else too of course, at the moment there are three mirrors that comes and goes.
|>>||MadHax 2may2012(we)17:35 No.3550 E |P18R15
|How about that command-line version? I have a US server that's up 24/7, but I can't do anything until the mirror stops trying to be a user app.|
|>>||Maharene 2may2012(we)23:38 No.3551 B |P19R16
I ran the mirror for a 24 hour period and counted 147 instances of "sent" files, though that's keeping in mind that my library is only 72 flashes strong and slowly growing based on what's popular on /f/. Only once did it have to reconnect itself in the middle of the night when I wasn't awake:
2may2012 03:25:27 Looks like swfchan might be down... Will keep trying.
2may2012 03:26:42 Server: Your mirror is up and running. Thanks!
2may2012 03:27:12 Reflecting 72 flashes on swfchan.
|>>||Rovencrone 3may2012(th)01:24 No.3552 F |P20R17
|So I started mirroring as well, and I'm looking forward to the auto downloading feature. When it does get implemented though, do you think it'd be possible to implement a sort of filter of what files to dl? ie by name/size or whatnot.|
|>>||Rovencrone 3may2012(th)01:39 No.3553 F |P21
|A bug I'm noticing is that current transfers always display 0.0% 0b/s, and simply disappear at completion. These values should update to display the download progress of the downloaders right?|
One more suggestion, could you add a Sent/Aborted counter to the gui somewhere?
|>>||!///SWFAnts #ADMIN# 3may2012(th)20:37 No.3554 SWF |P22R18
I was going to add the command-line feature in the previous version, but I forgot. Good news though since I've added it in today's 0.8.4 release. I hope it works well.
Sounds great. That mid-night reconnect was probably not a bug or anything, swfchan could have gotten overloaded for a brief moment.
I suppose a filter can be a good idea, but will it be that useful? What would you use as a file name filter for example?
That was indeed a bug, fixed it in today's update. Also included is such a counter that you were asking for.
|>>||Rovencrone 3may2012(th)23:57 No.3555 F |P23R19
Hmm, I suppose the name filter was something on the spur of the moment. however I suggest the file size limit mainly due to the fact that my upload bandwidth isn't the greatest, and due to that I feel like anything over 3-4 megs takes a bit too long for me to upload for the downloaders that they abort, thus it'd be kind of silly for me to host larger ones. Then again they're all rare and in between, but lots of music video flashes tend to fall in this category.
|>>||Rovencrone 4may2012(fr)00:33 No.3556 F |P24
|Btw, what exactly does "files sent to outside users" imply?|
|>>||!///SWFAnts #ADMIN# 4may2012(fr)06:19 No.3557 SWF |P25R20
Ah I see, well a filter wouldn't be hard to implement so sure I can include one when the day comes for the auto-download feature.
When someone requests a file their browser usually includes a referrer notice of which page they arrived from, if that referrer is not swfchan the request counts as an "outside user". It's there so that you'll be able to download files from yourself if you wish to without it changing your stats. Some very few people has however set their browsers to not include a referrer, then they will appear as "outside users" even though they did in fact come from swfchan.
|>>||Rovencrone 4may2012(fr)06:45 No.3558 F |P26R21
|Ah that clears things up. Also oh god I wasn't paying attention and had about 20 flashes saved there that were 404.swf. Cleaned them out but now I feel bad XD I suppose the size filter could be used to prevent the download of the 369kb flash, or just filter out 404.swf itself if the autodownloader runs into problems with the server.|
|>>||Anonymous 4may2012(fr)08:23 No.3559 A |P27R22
MD5 checksums could be used for this.
Also, as soon as the auto-download feature is enabled, I'll set up a mirror.
|>>||!///SWFAnts #ADMIN# 4may2012(fr)10:04 No.3560 SWF |P28R23
Which reminds me that I was going to include a setting that prints out a warning about duplicate files. I'll add it to the TODO list. Also there won't be a need to filter out the 404.swf flash, it will be impossible for the auto-downloader to get that. Actually I should do a overhaul sometime on the general download handler so that people would finally stop getting the 404 from time to time...
|>>||Anonymous 16may2012(we)09:28 No.3582 A |P29R24
|Oooh, very nice added features in the swfchan client! Now if only it had that auto-download feature... then I'd really want to set up a mirror.|
|>>||!///SWFAnts #ADMIN# 5jun2012(tu)01:58 No.3635 SWF |P30R25
|Version 1.0.0 of the swfchanMirror client was released today (well yesterday, a few hours ago). The auto-download feature is now available.|
|>>||Anonymous 5jun2012(tu)07:27 No.3643 A |P31R26
If you'd like, I could set up a mirror of the swfchanMirror program. Send me a email for further info. :)
|>>||!///SWFAnts #ADMIN# 5jun2012(tu)20:30 No.3644 SWF |P32R27
I'd like that, but I don't get why I should email for further info? What info? The mirror program is available for everyone and is completely voluntary, just head over to http://mirror.swfchan.com/ and grab the client if you wanna help out. There are detailed how-to instructions on how to set stuff up over there.
|>>||Anonymous 6jun2012(we)08:44 No.3647 A |P33R28
I meant mirror the program's archive file, not set up a instance of the program itself. Sorry I wasn't exactly clear on that. Hence why I said you should email me so I could set up a FTP account for you on my server.
Apologies for the miscommunication. :)
|>>||!///SWFAnts #ADMIN# 6jun2012(we)18:51 No.3648 SWF |P34R29
I see. A mirror for the program itself is not really needed since it's only about 100 KiB, but thanks anyway.
|>>||dollspit 7jun2012(th)03:57 No.3653 G |P35R30
|Port Forwarding issues|
So, I'm using a Netgear WNR2000v3 with firmware version V188.8.131.52.
I'm a little confused with the details of adding these open ports.
I go to the Port Forwarding/Port Triggering page and then click Add Custom Service. It has open dialogues for Service Name, External Starting Port, External Ending Port, Internal Starting Port, Internal Ending Port and Internal IP address. It also has a drop down menu for the protocol. I'm not sure where to put what information.
I'd love to be a part of this. Sorry I'm such a tech noob. Any help would be great!
|>>||Anonymous 7jun2012(th)07:15 No.3654 A |P36R31
|>>||!///SWFAnts #ADMIN# 7jun2012(th)19:33 No.3655 SWF |P37R32
Thanks for setting up a mirror! Here's what you need to do:
1. WinKey+R, type "cmd". When the console opens, type "ipconfig".
2. Look what your IP Address is. "192.168.0.3" or something similar.
3. Go to the page where you config your router, where you said you had been.
4. Add Custom Service (1 of 2):
> Service Name = swfchanMirrorUDP
> External Starting Port = 12345
> External Ending Port = 12345
> Internal Starting Port = 12345
> Internal Ending Port = 12345
> Internal IP address = "192.168.0.3" (just an example, put your PC's IP here)
> Protocol = UDP
5. Add Custom Service (2 of 2):
> Service Name = swfchanMirrorTCP
> External Starting Port = 54321
> External Ending Port = 54321
> Internal Starting Port = 54321
> Internal Ending Port = 54321
> Internal IP address = "192.168.0.3" (just an example, put your PC's IP here)
> Protocol = TCP
6. Go to Windows Firewall on the control panel (or your custom firewall if you use one).
7. Open port 12345 for UDP.
8. Open port 54321 for TCP.
9. If there's no clear option to open ports (depends on Windows version) you can add exceptions for "java.exe" and "javaw.exe" to access the Internet (these are located somewhere in "C:\Program Files\Java\<...>\bin\").
10. Start swfchanMirror.
|>>||dollspit 10jun2012(su)02:49 No.3694 G |P38R33
|>>||!///SWFAnts #ADMIN# 11jun2012(mo)03:00 No.3702 SWF |P39R34
Fantastic! I think I noticed your mirror being up yesterday.
|>>||Anonymous 8 11jun2012(mo)04:00 No.3707 H |P40R35
|So the server reckognises a flash by it's hash or md5 or whatever checksum, not by name, right ? So it doesn't matter what names I put them for sharing ? Because I often rename them (slightly).|
another question: Is there an option in INI file to minimize RAM usage ? (I know, this will make it harder for the HDDs).
|>>||!///SWFAnts #ADMIN# 11jun2012(mo)05:36 No.3712 SWF |P41R36
Yep, a flash is recognized by it's hash so it doesn't matter what name you have for it. The hash calculation doesn't follow any known algorithm.
There is no RAM control at the moment, other what Java's Virtual Machine provides (unlike machine code programs written in for example C, Java programs can not make use of all memory on the system unless the VM says so). A minimum of 250 MB must be accessible for swfchanMirror to run as a mirror however.
Right now I'm aware that the client has a memory "problem". There is no memory leak or anything like that, I just know that the program does take up quite a bit of memory. I'll try to reduce how much RAM it needs in future versions of swfchanMirror. It's possible that at the moment it keeps a bit more than it needs in memory.
I recommend giving the mirror at least 500 MB memory at the moment on v1.0.2 (note that this means that it will use up to maximum 500 MB only if it needs to, it will take less memory if it can), however you can run a mirror with the minimum 250 MB if you like to. The mirror doesn't actually require the extra memory unless your mirror becomes busy with lots of requests at the same time for example, then it's a good thing if the program is allowed to allocate those extra bytes in memory (if you see any OutOfMemoryErrors the VM needs to be allowed more memory). If the mirror load goes down the program will take less RAM again, the Java VM monitors your system and performs garbage collections as it is needed.
Use the Xmx VM flag to set the allowed memory for the program. Usually (when not defined) the VM sets the max memory to a percentage of the system's total RAM. The first time you run swfchanMirror.jar a bat-file will be created next to the ini file, if you run it it will launch swfchanMirror.jar with an allowed max memory of 500 MB. You can edit that bat file to set the max memory to 250 if you'd like.
I don't think I will lower the required max below 250 MB, however I will later try to make the program want to be as small as possible. Though memory handling is difficult in Java since it's done by the VM, it handles clearing of memory automatically whenever it want to. I can suggest it to run the garbage collector in my code but it might not do it until much later. Maybe it won't do it until when it notices that other applications may actually need the memory. After all, a full RAM is no problem as long as no other program needs to use any of it. And as long as there is some RAM to clear for other programs to use as soon as they do need it there's no issue then either.
Try out running a mirror, if it doesn't affect your other programs and if you don't get any OutOfMemoryErrors in the log then there's nothing to worry about. And if you do run into difficulties you can always stop mirroring at any time, no sweat. :)
|>>||Anonymous 8 11jun2012(mo)07:47 No.3714 H |P42R37
|Unfortunately I won't be able to share, as I can't get incoming packets, neither UDP, nor TCP. My ISP uses NAT. Torrent clients would use "UDP hole punching", or whatever it is called, making UDP traffic possible, but apparently the mirror server does not. Still the problem with incoming TCP would remain though.|
ps: I don't use router.
|>>||!///SWFAnts #ADMIN# 11jun2012(mo)15:04 No.3715 SWF |P43R38
|I'm sorry to hear that you have such a bad ISP. Would you mind telling me your location, are you in China?|
|>>||Anonymous 8 11jun2012(mo)17:30 No.3719 H |P44R39
I wouldn't call my ISP bad, especially with the speeds I have. Regular users don't have to set up a server every day, after all.
The NAT type they use is "one-to-many" (looked it in wiki, so there is no misunderstanding). It makes incoming TCP connections impossible, probably because they didn't bother configuring correctly the port forwarding on their side. Several years ago I asked them to do it, but got only some vague reply, which could be translated as NO :) Then uTorrent presented UTP protocol (a software implementation of TCP, using UDP), which came with the UDP hole punching, this making it possible to receive incoming "connections", and my problem was solved (torrents are pretty much the only thing I need a two-way connectivity for). Since then - around 3 years ago - I completely had forgotten about this, and now it all came back to me.
And just to make it clear again: I can receive TCP packets of course, but only if my PC initiated the TCP connection.
In theory swfchan mirror server can be made so it works with this type of NAT- it can use both TCP and UDP to communcate to swfchan, and swfchan can use one of the NAT traversal techniques using UDP to communcate with the mirror server. When users wants to d/l from me and clicks on a link, swfchan can send me udp message "hey, this IP wants to get this flash (hash here) from you", and upon receiveing it the mirror server initiates the TCP connection needed.
This is just an example. Of course you don't have to code this, as I'm an isolated case, and implementing NAT traversal techniques may prove to be difficult. Just keep it in mind, if more people like me (behind NAT) appear.
Oh, and Bulgaria here, not China :)
|>>||!///SWFAnts #ADMIN# 11jun2012(mo)17:46 No.3720 SWF |P45R40
I understand what you are suggesting but it can't be done, you can't initiate communication with a strangers browser. Browsers don't accept incoming connections, they must always send a request to the server first. So setting up a mirror is impossible if connections can't be received.
Out of curiosity, what download and upload speeds do you manage to get over there downloading for example a popular movie or something from a public torrent?
|>>||Anonymous 8 11jun2012(mo)18:41 No.3721 H |P46R41
I see, then nothing can be done. Let's hope soon IPv6 will reach the end user, and problems like this will be thing of the past.
here's a speedtest I've just done: http://www.speedtest.net/result/2002863963.png
The bandwidth I pay for is 20 MBit/s, actual speed for bulgarian servers / sites is capped at 4 MiB/s, for foreign ones at 2MiB/s, and I hit these speeds when torrenting popular content. Upload speed is rather inconsistent, I've seen 2 MiB/s upload, and sometimes it doesn't reach even 50 KB/s, seems to depend on different countries, and as I observed, on weather.
All in all, I'm happy, as you see they give me more than advertised :)
|>>||!///SWFAnts #ADMIN# 11jun2012(mo)20:13 No.3722 SWF |P47R42
|>>||Anonymous 16jun2012(sa)01:40 No.3744 I |P48R43
I just downloaded & tested the mirroring software, copied a couple of random flashes to a box running it, everything seems to work nicely - for those few test flashes.
How do I get it to automatically download popular flashes from swfchan servers for mirroring?
The following (default) settings didn't seem to do anything:
>0801) Auto-mirror new swf-files with at least this many views/day (from newcommer list): 5
>0802) Auto-mirror popular swf-files with at least this many views/day (from main list): 5
>0803) Auto-mirror popular old swf-files with at least this many views/day (from main list): 30
I was thinking I could dedicate roughly 8-10 Mbit/s of my upload for swfchan mirroring around the clock, with above average routing/ping for european users. I can't spare much cpu cycles though, but this shouldn't be a problem if the mirror software works properly. ;)
|>>||Anonymous 16jun2012(sa)02:24 No.3746 I |P49
Nevermind that, I didn't bother reading the mirroring guide, just grabbed the rar and tested how it runs.
You might want to include this part somewhere near the auto-mirroring settings:
>There are three requirements that needs to be met before the server starts to send your mirror new/popular files:
>1. You need to share at least 10 flashes that exists on swfchan.
>2. You need to have served at least 20 flashes to swfchan users in the last two days.
>3. Your mirror must have been online for 4 hours straight.
|>>||Anonymous 16jun2012(sa)05:44 No.3747 I |P50
Aaand some initial observations about the mirroring service:
* When requesting a flash from swfchan, the Available mirrors -table shows average speed of the mirrors. IMHO, this is misleading for users. Users with slow connections cause slow average speeds, despite the mirror having much more bandwidth available. This leads to users with more bandwidth always preferring swfchan instead of a mirror. Which in turn lowers the mirrors' reported average speeds even more. A positive feedback cycle. Average speeds are more limited by users than hosts. Not a very useful metric when your aim is to divert traffic to the mirrors.
For example - a user with an average DSL/cable connection (maybe around (4 Mb/s download bandwidth?) sees mirror Anonymous: 200 kB/s average? "That's slow, better use swfchan instead", although the mirror has 2.19 MB/s top speed and the capability to serve files much faster than the average suggests. Hell, I never used any mirrors myself because of the slow reported speeds. Also, many less tech-savvy users tend to confuse bits and bytes, so it might be better to use bits instead of bytes. Would an average joe rather use 2.19 MB/s or 17.52 Mb/s mirror? Which would be more informative?
* No proper logging. Should someone need or want to monitor swfchanmirror's traffic activity in-depth, they'll need 3rd party software. An option for advanced logging (those 2-char ip identifiers aren't helpful!) would be nice to bring swfchanmirror more in line with other http servers.
* Far future suggestion: To attract more mirror hosts, make a simple windows installer that configures the server and starts it up as a service. Next-next-next. Maybe advertise it on the front page. Can't think of a better way to get many people to help.
Also, do you have any statistics about how much load your mirrors take away from swfchan.org?
|>>||Anonymous 16jun2012(sa)08:24 No.3748 I |P51
And the story continues. After only a few hours of test hosting, java started shitting: http://i.imgur.com/k6SGx.png
The server was started with default parameters, ie. directly from the .jar.
So I thought, maybe it's the GUI. Restarted in console mode, didn't help much. Disk IO is fine, RAM usage is fine. Taking up an entire CPU core intermittently isn't fine. This was with 32-bit Java SE 6 U 24, by the way.
Next logical step: upgraded VM to latest version, Java SE 7 U5. Still getting ridiculous CPU usage spikes, but their pattern did change. Java GC related problem?
I'll keep the mirror running for now, unless it melts my server box. Any ideas would be appreciated.
|>>||Anonymous 16jun2012(sa)10:01 No.3749 A |P52R44
MackanZoor says he still needs to work on resource allocation with the program; he's currently focusing on adding more features.
Also, I second the installer idea; I could compile a installer version for Windows, but I'm not exactly sure how I'd come up with a way to make it run as a service. Would take some research.
If anybody shows more interest, I'd be willing to package every new version of swfchanMirror into a installer for Windows users.
|>>||Anonymous 16jun2012(sa)10:24 No.3750 I |P53R45
I managed to get it running in a bit more calm manner with JDK and -server -Xmx1024M -Xms1024M.
CPU usage is not tied to the server's activity, so I'm assuming it's the java VM's activity, running garbage collection probably. Giving it 1 GB min/max memory seems to keep it calm for a while.
About running it as a service, that's entirely optional, but here's a good resource on that if you're interested:
Simply adding a shortcut to all users' startup folder would be enough. Only real drawback would be that the server won't run when no one's logged in.
|>>||Anonymous 16jun2012(sa)11:58 No.3751 I |P54
Jesus christ how horrifying http://i.imgur.com/JPosW.png
One of the threads in javaw.exe is occasionally (almost constantly with default settings) eating over 3G cpu cycles per second. This is no server, this is a stress test. D:
Also, the intermittent cpu spikes are becoming more frequent as the actual memory usage is approaching 1GB. This shit doesn't work unless the mirror software is rebooted every 2-3 hours.
Any ideas how to tune java's GC so it isn't absolutely horrible?
|>>||!///SWFAnts #ADMIN# 16jun2012(sa)18:03 No.3753 SWF |P55R46
|Thanks a lot for setting up a mirror, I appreciate it!|
>I was thinking I could dedicate roughly 8-10 Mbit/s of my upload for swfchan mirroring around the clock
>I can't spare much cpu cycles though
>You might want to include this part somewhere near the auto-mirroring settings
I figured people that were interested in that would read the "auto download"-section of the mirror page on the site. Especially since it contains a numbered list, those usually draw attention on webpages. But you are right; I should mention that there are a few requirements in the settings file. I'll add it to my TODO list.
>the Available mirrors -table shows average speed of the mirrors. IMHO, this is misleading for users
Although you are right it doesn't have that much of a difference about the traffic diverted from the site. Most flash files are viewed using the "embed" option, then the server automatically assigns a mirror to download the flash from (if available). The mirror with the least current connections are picked, with a chance to use the swfchan server too so that the mirror(s) don't carry the whole burden. I've added a note to the non-embed download page that informs the user that downloads from mirrors will probably go faster than if they download from the swfchan-server.
I have plans to change a bit how the average and top speed is calculated. I'm thinking that the average speed will be the average of the middle 60% of all connections (when sorted by speed). This way the few lowest and few highest speeds will not drag down/up the average too much. It becomes a near-average that should give a more realistic image of the typical speed. The top speed will become the average of the top 10% of all connections, so that not a single super-high speed will be reported (can be misleading).
>many less tech-savvy users tend to confuse bits and bytes, so it might be better to use bits instead of bytes
I've never liked the Mb/s measurement (or how its more commonly written: Mbps). Talk about confusing. 10 Mb/s = A top speed of 1.22 MiB/s (10000 bits / 8 / 1024 = 1.220703125 MiB). It's better to show the speed in MiB per second because that's how people really measure speed of downloads. It's the speed that browsers report that they are downloading in.
I also always use the correct MiB term instead of the incorrect MB term (even if Windows show file sizes as "MB" they are actually calculated by dividing by 1024, which is MiB (if MB it should be divided by 1000)). This often causes major confusion when people buy a new HDD labelled for example 200 GB. When they come home and format it it turns out as 190.7 GB and they feel ripped off. But the size is actually 190.7 GiB (200000000/1024/1024) which equals 200 GB (200000000/1000/1000). I hope in Windows 8 file sizes show up as KiB/MiB/GiB/TiB so this confusion can finally start to die out.
>No proper logging.
I've intentionally left out stuff like user's IP addresses from the log, so that those that run mirrors shouldn't be able to easily keep track of other people's business on swfchan. The two-character IP identifier was added so that you should be able to see when a single transfer starts/finishes on a short-term, and also if a person is using your mirror to download a lot of files. I won't add much more than that so that mirrors can start to track user behaviours on the long-term.
>make a simple windows installer
A big point of Java is to not make platform-specific tools such as installer executables. Besides, is it really needed? It's one file that needs to get "installed", personally I always prefer archives that you extract rather than installers that you don't really know what they are doing. I know that making things as simple as possible is good (next next next) but those that are interested in setting up a mirror should be able to start a jar file and edit a ini file. Maybe add the thing to the startup folder so it starts every time the computer start. If the program was meant for the everyday joe I can understand why a installer would be needed but most common folks won't download a server program. If anything I might add buttons inside the GUI later that allows you to edit the settings or make the program auto-start upon boot.
>Maybe advertise it on the front page.
Making a site-wide broadcast upon every new version of the mirror program. I've realized that some otherwise interested will miss it though, especially when the mirror updates get less frequent. Sometime later I will add a "site tip" thing at the bottom of every page with random "did you know" stuff about swfchan (for example "Did you know you can make your own flash loops here on swfchan?"). I'll mention the swfchanMirror there so that there's a higher chance people will discover it and might wanna help out.
>do you have any statistics about how much load your mirrors take away
Not really any numbers, but the network load average has gone down from 90% to 70%. It still hits 100% sometimes but that just means that more bandwidth can be given to fewer connections. Several hundred GiB of bandwidth has been saved.
No, Java is great! Not only is it convenient not having to compile the program into machine code so it works on all platforms, it's very easy and comfortable to program in. I actually consider programming in Java fun, everything makes so much sense. And Java has a thing called just-in-time-compilation which compiles its byte code into machine code as the program is running, which means that the programs executes just as fast as any program written on a lower level (C for example). There are ways of executing C-code from inside Java very easily using Java Native Access (JNA) so you can do all the things you normally can't do in Java (make program hooks that listen to key presses in other programs for example - it's against Java's security model but can be done with C). The memory handling in Java is fantastic with the garbage collection, it makes memory-leaks almost impossible (there's a big difference between leaking memory and a program that requires much memory).
As for the CPU problem you are having, it's very strange. What operating system do you have? I've tried having mirrors up for several days on both Windows Server 2003 (2.5 year old CPU, used -Xmx350m) and Windows XP (5 year old CPU, used -Xmx700m). Both using Java Runtime Environment v6 and both with CPUs that weren't very powerful even when they were new. The process CPU usage is almost always 0%, jumps up to 2% when a connection is accepted and then goes back to 0% after a second. Once a minute there may be a spike of 10-15% CPU usage lasting about a second. Both test mirrors were on heavy load serving more than 6000 files in 24 hours. I got a few out-of-memory errors now and then (I'll try to reduce the memory requirement in a later version) but the garbage collection didn't really make the CPU sweat anytime and the servers kept going. The GC must have run a lot during the day or else the log would be flooded with out-of-memory errors but there were only a couple a few times.
Are you sure it's not another program that causes the CPU load? Have a look at the processes individually. Do you run another Java program at the same time? I think several Java programs might share the same Virual Machine process. Hmm... maybe you have a program up that tries to monitor or interfere with the Java program? Bandwidth limiter or some sort of controller perhaps? Do you launch swfchanMirror from within another process?
Thanks for the feedback! Always welcome. I hope you won't take down the mirror since it seems to be doing much good with great speed. :)
|>>||Anonymous 17jun2012(su)00:16 No.3754 J |P56R47
First of all, the problem really seems to be related to Java's garbage collection, more precisely calls to System.gc(). See this: http://i.imgur.com/WqP2Y.png - guess what happened in the last case when the server actually filled that gigabyte? All the CPU usage graphs are from the java process running _only_ swfchanMirror. I don't have any other Java stuff running on that box. I am using various versions of Sun/Oracle's Hotspot JVM. If you know any better JVMs, let me know!
I am aware of the KiB / kB distinction. But I have to admit, I use them interchangeably when precise numbers aren't important - bad habit. I prefer using the base-10 bits and bytes since they still are the leading standard. And bits vs bytes, I guess that's a cultural preference - around here ISPs and us IT guys always use megabits per second when talking about bandwidths, ie. connection speeds.
The whole windows installer & frontpage idea was to make mirroring as easily approachable as possible for as many users as possible. I'm sure you understand the potential benefit of that, even though it goes against the glorious portability of Java. :)
Now, I am not a programmer and I don't want to turn this into argument whether java is good or bad, but I can't host a permanent mirror because java gc uses ridiculous amounts of processing power, ie. java sucks.
I wonder if I can rig application recycling to restart the java process before its working set memory reaches its Xmx value...
|>>||!///SWFAnts #ADMIN# 17jun2012(su)02:38 No.3755 SWF |P57R48
|Okay, what I've learned is that I should never use System.gc() calls. Seems like they are a good idea on paper but works badly in practise. I don't know why they causes your CPU to go up so much but didn't in my tests. Guess there are many factors. Hotspot is the official Java VM I think so you shouldn't have used a different VM than me either. I will remove the GC calls. If possible I'll release a new version of swfchanMirror tomorrow, probably just a minor edit with the GC calls removed and not much difference otherwise. If nothing else we will eliminate one possible cause of the problem.|
>guess what happened in the last case when the server actually filled
Either the mirror crashed or (more likely) the CPU went up a lot since a "natural" garbage collection must have been called by the VM? Perhaps the CPU went up and the program then crashed since it maybe couldn't free enough memory in time?
Btw I assume that you are using Windows since you are using Process Explorer but which version of Windows exactly? It's good to know.
>around here ISPs and us IT guys always use megabits per second when talking about bandwidths
Yeah same here and probably around the world, when it comes to describing bandwidth. And I do that too, when describing the connection capacity. But people usually don't talk in Mbps when they are talking download speeds, I can't think of one single program that says something like "you are downloading bla.rar in 33 Mbps" instead of saying "3.9 MiB/s" (or rather the incorrect "3.9 MB/s", which really still is MiB).
If possible, check back in 24 hours and see if there's a new GC-free version of the mirror. I'd soothe my mind if I knew the issue ceases.
|>>||Anonymous 17jun2012(su)03:47 No.3756 I |P58R49
>Either the mirror crashed or (more likely) the CPU went up a lot since a "natural" garbage collection must have been called by the VM
Yeah, GC kicked in (does the "natural" GC use same system.gc() calls?) and the mirror software starting eating up all the CPU it could get. Some out of memory errors in the log, but it didn't crash! I was honestly surprised that it kept working.
>which version of Windows exactly?
32-bit Windows 7, OEM license. I'd rather use 2003 server, but don't have any extra licenses around. :(
>I have plans to change a bit how the average and top speed is calculated.
Forgot to mention earlier, these sound great!
>talking download speeds
Ah, this is where we see things differently - I was trying to think what would be most informative for users when they see a mirror in the list - The mirror's bandwidth or average download speed from that mirror?
In any case, it doesn't matter as much as I thought since embed downloads choose a mirror automatically.
Perhaps you could make direct swf downloads choose a mirror automatically in the future?
>If possible, check back in 24 hours
I will. I have extra bandwidth, I want to help swfchan, and frankly, this project is very interesting! I don't think I've ever seen a website do something like this. I would be greatly interested in any mirror statistics and performance numbers you are willing to release.
|>>||!///SWFAnts #ADMIN# 18jun2012(mo)00:00 No.3758 SWF |P59R50
|The new version is out! I think I over did it a bit though. Turned out as a major release instead of a minor one.|
It's too early to tell for sure but so far it looks like I managed to get the program to take up much less RAM, however it increased the CPU usage a little bit for me which I have to investigate for the next release (suspect it's the more exact upload speed that is the cause). But the CPU usage only increased by a single percentage point or two so it's not that big a deal, but less resource requirement is always better so I'll see later if it can be reduced.
>does the "natural" GC use same system.gc() calls?
There's only one kind of GC. The System.gc() call is just something that tells the VM that "now is a good time to pick up trash". The VM can ignore it completely or go ahead and do it if it thinks it's a good idea. Problem was that your VM for whatever reason thought it was always a good idea.
>but it didn't crash
That's the power of Java! ;D Nah, only the thread that requested the memory crashes, the rest of the program (in different threads) that already have memory assigned goes on as usual without even knowing that a thread died from memory problem.
>32-bit Windows 7, OEM license
I'll try to make a test mirror on Windows 7 sometime in the future. I installed Win7 on a old laptop a few days ago, maybe I could test it on that if I get my hands on it again. It has hardware dating at least 10 years so it would be interesting!
>The mirror's bandwidth
But it wouldn't really represent the mirror's bandwitdh since it all depends on where the users are in the world, better to call it download speeds because that's what they are. :) Potato potato, tomato tomato.
>make direct swf downloads choose a mirror automatically
Maybe, but as it is now the user gets presented with a bunch of links and the top one is the one that the server "recommends" (the mirror with the least number of connections at the moment). At the moment that's probably the most logical way of giving the download options, since the server has no way of really knowing which mirror is best for a user.
If I ever implement some kind of system that can compare a user's and a mirror's geographical location and pick out the one that is closest to the user I will definitely have the server specifically tell the user that it should download his file from there.
>I don't think I've ever seen a website do something like this
Hehe, me neither. That's a big reason why I'm putting time into the project actually, it's a kind of experiment. Something new for me to try. And when you try something new you always learn something new!
|>>||Anonymous 18jun2012(mo)05:00 No.3761 J |P60R51
Here we go again. Running the new version with JDK -server -Xmx1024M -Xms1024M (I really really prefer the hotspot server VM) and so far, after 1 hour, its behavior is mostly as you described.
After the initial swf request rush, CPU usage settled to be a little bit higher than in r6. Still, it's a very low load on any half-modern x86 CPU.
Working set size seems to be growing slowly, its at 900 M and hasn't reached the limit yet.
IO read bytes seem to be climbing up slower and random disk IO spikes are clearly less frequent.
> (suspect it's the more exact upload speed that is the cause)
Does this happen with the console option? I don't need real-time statistics like that nor on-screen log-file buffer, so I might as well run the mirror completely hidden with javaw and the console option - once I'm confident about its resource usage and stability.
Hey, I just found out about jconsole. What a wonderful tool!
|>>||!///SWFAnts #ADMIN# 18jun2012(mo)18:22 No.3765 SWF |P61R52
|I had to put together version 1.1.1 today real quick since I noticed that the average speeds of the mirrors were decreasing steadily since the last release. That should be fixed now. Also took the opportunity to increase the URL timeout since if option 0508 is enabled to make URLs private there is little need for a short timeout since links can't be shared anyway.|
>I really really prefer the hotspot server VM
As does Oracle (or Sun) since it's the official VM in Java. I have "Java HotSpot(TM) Client VM version 19.1-b02" on my development machine according to JConsole.
Thanks about the tip about JConsole btw, I've actually never used it before. It's bound to be very useful in the future. It even has a "Perform GC" button!
>Does this happen with the console option?
Yep, the only difference with the console option that no GUI operations are made. The upload speed calculations are the same in the background.
The changes I made today should have lowered the CPU usage that was introduced in the previous version BUT since I fixed the low transfers speeds the CPU usage may actually increase since the mirror could become more busy at serving files. This might cause it to fill the RAM quicker too, kicking in the first GC a bit sooner.
Could you remove the -Xms1024M option from your VM a little while and try to describe how long it takes for the memory usage to hit around 900-1000 MB? I know that making the VM start out with a lot of reserved RAM makes it a bit less CPU needy when it is time to use more memory, you can restore using -Xms1024M later. I was just curious how long it takes for the mirror to reach the set max memory on your system.
|>>||Anonymous 19jun2012(tu)06:25 No.3781 J |P62R53
>since it's the official VM in Java.
Thing is, Hotspot is actually 2 different VMs. Running anything with default settings uses the Client VM (optimized for memory usage, quick startup) while -server uses Server VM (optimized for long-term performance)
>Could you remove the -Xms1024M option
Do not want. We both know this only leads to increased CPU load in an endless cycle... I prefer to keep Xms and Xmx the same.
>how long it takes for the memory usage to hit around 900-1000 MB
I can still help you with that. JConsole shows that swfchanmirror r7 performed its first major GC around the first hour mark, dropping heap size from ~900 M to ~600 M. The collection wasn't a major CPU spike and the mirror kept running smoothly afterwards as well.
I also learned that Process Explorer's working set size != actual java heap size. :(
I'll check how the new version performs.
|>>||Anonymous 19jun2012(tu)08:10 No.3782 J |P63
Well, I just started version 1.1.1.
It seems to be eat memory faster and also perform GC more aggressivily (one every 3 minutes).
GCs became larger but less frequent as heap size was getting closer to its limit.
After about 1h 40min, heap size had reached 900 M with a GC happening every 10-15 minutes.
CPU load averages at about 4-5%, no much difference from the last version.
PS. I don't know anything about all these java memory pools, so I just checked the heap memory usage chart.
|>>||!///SWFAnts #ADMIN# 19jun2012(tu)14:41 No.3783 SWF |P64R54
|Alright thanks. I had forgotten about the server option. I don't think the server is included in a standard java runtime environment, one would have to get a java development kit for that. Too bad, otherwise I'd be able to put -server in the bat file that's created when the mirror is run for the first time.|
My test mirror ran into a issue after about 14 hours of uptime, causing the CPU load to increase a lot and no new connections to be accepted. It's pretty possible it was because of too many transfer requests. I'll have to look into that. It also appears like the mirror has started to reflect 0 files every now and then, I know the cause of that though so it won't be a problem to fix. Hm, actually that might just be the reason as to why the CPU and memory usage has become more aggressive.
|>>||!///SWFAnts #ADMIN# 20jun2012(we)05:32 No.3786 SWF |P65
|Put some effort into increasing performance today and now version 1.2.0 is out. There's a huge difference, memory is used more efficient and the CPU load is lower. It's been pretty fun to track down and fix the previous flaws in the code but I'm sure something will pop up that I have to give further attention. It'll probably be something that has to do with what happens if the program actually does run out of memory (due to a sudden big increase in concurrent connections). If I would have to guess I'd guess that the next version's main implementation will be some sort of defence mechanism against OutOfMemoryExceptions.|
It's hard to overstate my satisfaction at the moment. I think I'll treat myself to some pancakes and ignore that it's pretty much morning over here now.
I hope some of you guys will try out the new mirror. I want to hear if it runs any better on your system now.
|>>||Anonymous 26jun2012(tu)21:31 No.3821 I |P66R55
|After some 3 days of uptime, swfchanmirror is averaging less than 4% CPU load, which would be fine (although quite high for the simple function the software performs), but still - all the CPU usage comes as massive spikes from large GCs. Without them, performance would be excellent.|
A shame, really. If not for Java, it could perform the same function while using only a small fraction of resources. It's not your fault; Your software performs very well, JVM does not.
I guess I'll try Garbage First before giving up on Java. It should have exactly the kind of performance upgrade I'm looking for.
>main objectives for the new collector:
>The first objective is consistent low pauses over time.
>The second objective is to avoid, as much as possible, having a full GC.
|>>||!///SWFAnts #ADMIN# 27jun2012(we)05:06 No.3822 SWF |P67R56
|Yes the high CPU usage of the garbage collection procedure is a pity... But it only lasts for one to three seconds at most. Even if it means that Java has a small drawback the fact that memory management is taken care of automatically more than makes up for it. That's something that otherwise can take years to fully master in complex projects. Besides, the drawback is only more noticable in this project since the mirror rotates a lot of files in memory. In most situations I think the GC issue wouldn't even be there.|
One can always hope that future VM implements better GC procedures. Although even if that doesn't happen CPUs will beome faster so the drawback will become more insignificant with time.
Let me know how it goes with using the G1 garbage collector, sounds interesting. Maybe it'll make all the differance in the world.
|>>||!///SWFAnts #ADMIN# 29jun2012(fr)04:00 No.3827 SWF |P68
|Discovered that something just caused a deadlock in my test mirror. This means that two or more threads are waiting for each other to finish and will thus wait forever. Not sure how it happened but it seems to be Swing related (when updating the text area in the window). I will investigate this as soon as time lets me, just wanted to note down that I'm aware of the issue.|
Those of you that are running a mirror at the moment please check it, if it is consuming unusually much CPU it means that a deadlock has happened and it is not killing off threads like it should. You should manually kill the mirror process (the one that is taking up much CPU) and restart the mirror. This thing should be rare to happen so don't be afraid to restart the mirror. From the log it doesn't look like this is related to a out-of-memory issue but rather some kind of deadlock inside Swing itself (code I have not written). Not very likely though, there's probably something I can do to prevent this. Perhaps a uncaught exception that causes a thread lock to not release (though that shouldn't happen, hm). I'll look into it tomorrow or during the weekend.
|>>||!///SWFAnts #ADMIN# 29jun2012(fr)04:01 No.3828 SWF |P69
|Wonder why I always discover these problems so damn late each time.|
|>>||!///SWFAnts #ADMIN# 8jul2012(su)17:45 No.3874 SWF |P70
|All issues I had found in swfchanMirror has now been fixed in the 1.4.0 release.|
|>>||!///SWFAnts #ADMIN# 27sep2012(th)22:18 No.4635 SWF |P71
It's been pretty dead for several months now and no interest has been shown so I