View unanswered posts | View active topics It is currently Sun Jun 24, 2018 2:20 pm



Reply to topic  [ 11 posts ]  Go to page 1, 2  Next
 Check Content directory for new files while running 
Author Message
Junior Member

Joined: Fri Dec 16, 2011 12:06 am
Posts: 11
Medals: 9
RadioBot Owner (1) Feature Requester (2)
Longevity (6)
IRCBot Version I Use: v5
IRCBot Platform: FreeBSD
Post Check Content directory for new files while running
Having a way to check just new files (periodically or by command), as opposed to reloading the entire directory would be useful if somebody wants to include a file uploader on their site for users to upload (straight to the content directory).

autodj-reload causes the stream to disconnect, so perhaps just having that run independently somehow would be sufficient.


Fri Dec 16, 2011 7:15 am
Profile
Site Admin
User avatar

Joined: Thu Oct 16, 2008 1:58 pm
Posts: 451
Medals: 18
Administrator (1) Plugin Developer (1)
RadioBot Owner (1) Helper (1)
Longevity (14)
IRCBot Version I Use: v5
IRCBot Platform: Linux/Ubuntu
Post Re: Check Content directory for new files while running
To find new files it has to rescan the whole directory (just like any program). There isn't currently a timed option to rescan files but you can use !autodj-reload to do it manually any time; I should be able to add a timed option as well. Until then, one trick you can do is create an entry in your scheduler.conf to request a file that doesn't exist; that would rescan for files however often you made the request fire off.

The stream can disconnect during a rescan if the current song stops playing while the scan is in progress, but it doesn't force the stream to disconnect as a rule.

_________________
You can also contact me via email or instant messenger, links are at the bottom of this post.


Fri Dec 16, 2011 9:26 am
Profile YIM WWW
Junior Member

Joined: Fri Dec 16, 2011 12:06 am
Posts: 11
Medals: 9
RadioBot Owner (1) Feature Requester (2)
Longevity (6)
IRCBot Version I Use: v5
IRCBot Platform: FreeBSD
Post Re: Check Content directory for new files while running
Ah, good to know about the scheduler trick. I just tried running !autodj-reload a few times, I didn't realize it was a coincidence that it was disconnecting before. D'oh. I'm running a few thousand files on a with Content pointing to a 20mbit nfs share, so yeah. The reload in ircbot5 is fast nonetheless. :D


Fri Dec 16, 2011 9:38 am
Profile
Site Admin
User avatar

Joined: Thu Oct 16, 2008 1:58 pm
Posts: 451
Medals: 18
Administrator (1) Plugin Developer (1)
RadioBot Owner (1) Helper (1)
Longevity (14)
IRCBot Version I Use: v5
IRCBot Platform: Linux/Ubuntu
Post Re: Check Content directory for new files while running
How long does it usually take to reload over NFS? I've always wondered how slow it would be scanning a remote filesystem.

_________________
You can also contact me via email or instant messenger, links are at the bottom of this post.


Fri Dec 16, 2011 10:22 am
Profile YIM WWW
Junior Member

Joined: Fri Dec 16, 2011 12:06 am
Posts: 11
Medals: 9
RadioBot Owner (1) Feature Requester (2)
Longevity (6)
IRCBot Version I Use: v5
IRCBot Platform: FreeBSD
Post Re: Check Content directory for new files while running
It depends, on ircbot5, on a first load, it's 5 minutes every 1k files. Then I can do !autodj-reload and it'll be less than a minute total. Over the summer, it was more like 45 minutes every startup (for 40k files) on ircbot4 thanks to mysql queue. I'll be able to post a more coherent benchmark in a couple days after I organize/transfer some files.


Fri Dec 16, 2011 10:46 am
Profile
Site Admin
User avatar

Joined: Thu Oct 16, 2008 1:58 pm
Posts: 451
Medals: 18
Administrator (1) Plugin Developer (1)
RadioBot Owner (1) Helper (1)
Longevity (14)
IRCBot Version I Use: v5
IRCBot Platform: Linux/Ubuntu
Post Re: Check Content directory for new files while running
As part of the file scan it calls stat() on each file to get it's last modified time and file size to determine if the bot should re-read it's ID3 tags, I imagine that would take a lot of time with thousands of files over a network link. I could probably make an option to read tags for new files only and that *should* make it go a lot faster, at the cost of if you update any of your ID3 tags or files the metadata won't be updated in the bot unless you do a full re-scan.

_________________
You can also contact me via email or instant messenger, links are at the bottom of this post.


Fri Dec 16, 2011 11:37 am
Profile YIM WWW
Junior Member

Joined: Fri Dec 16, 2011 12:06 am
Posts: 11
Medals: 9
RadioBot Owner (1) Feature Requester (2)
Longevity (6)
IRCBot Version I Use: v5
IRCBot Platform: FreeBSD
Post Re: Check Content directory for new files while running
Ah, that sounds great.


Fri Dec 16, 2011 5:55 pm
Profile
Junior Member

Joined: Fri Dec 16, 2011 12:06 am
Posts: 11
Medals: 9
RadioBot Owner (1) Feature Requester (2)
Longevity (6)
IRCBot Version I Use: v5
IRCBot Platform: FreeBSD
Post Re: Check Content directory for new files while running
Alright, here are the benchmarks.

With queue_mysql and nfs, after wiping ircbot.db and the IRCBot mysql table, file checking for 31827 files (with metadata) takes 3 hours 16 minutes. It's about the same as if it checked locally with queue_mysql, while it's faster (locally and nfs) with queue_memory. Restarting IRCBot with 3k files added took 32 minutes, autodj-reload right after that took 28 minutes.

Quote:
[15:14:32] <JAM> autodj-reload
[15:14:32] <Touhou_Radio> Reloading schedule...
[15:14:32] <Touhou_Radio> Beginning re-queue...
[15:42:19] <Touhou_Radio> Re-queue complete! (Songs: 38543)


Fri Dec 23, 2011 4:43 pm
Profile
Site Admin
User avatar

Joined: Thu Oct 16, 2008 1:58 pm
Posts: 451
Medals: 18
Administrator (1) Plugin Developer (1)
RadioBot Owner (1) Helper (1)
Longevity (14)
IRCBot Version I Use: v5
IRCBot Platform: Linux/Ubuntu
Post Re: Check Content directory for new files while running
A new build is up now with the new OnlyScanNewFiles option.

It goes in AutoDJ/Options:
OnlyScanNewFiles 1

That will remove the extra stat() call and maybe make it faster a bit? (I tested on a local hard drive with 5,000 files and the re-scans finished in 8 seconds with or without the setting, so I'm not sure how much help it will be.)

_________________
You can also contact me via email or instant messenger, links are at the bottom of this post.


Fri Dec 23, 2011 5:08 pm
Profile YIM WWW
Junior Member

Joined: Fri Dec 16, 2011 12:06 am
Posts: 11
Medals: 9
RadioBot Owner (1) Feature Requester (2)
Longevity (6)
IRCBot Version I Use: v5
IRCBot Platform: FreeBSD
Post Re: Check Content directory for new files while running
Awesome, thanks.


Fri Dec 23, 2011 5:12 pm
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 11 posts ]  Go to page 1, 2  Next

Who is online

Users browsing this forum: No registered users and 0 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.