| |
| |
| |
|
Page: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Comments:
<0> Ukyo: i dont know a better way of caching besides the idea i gave above
<1> and luckily for me
<1> these files dont 'update'
<1> so the cacheing server doesnt have to check for updates
<1> just nuke them after X period of no downloads
<0> you just need a timestamp check on the local copy.. no need for diffing via a md5 checksum or the like
<2> zircu: Was that for me?
<1> which processes and handles faster.. readfile() for using fopen and using a while to dump the file
<0> CWhiz: no.. for Ukyo
<2> ok
<1> eh?\
<1> i dont need to diff
<1> CWhiz does
<1> my files dont ever change
<1> once uploaded. they are there to stay
<0> Ukyo: well, at minimum you need a filesize check, what if the copy you have is bad in the cache?
<1> when the cache server copies it, it does a hash check
<1> before setting it active on the cache sql row
<0> Ukyo: ok... perfect
<1> the nifty thing about my design for the filesystem clusters
<1> is as i said, at any time i can just toss A/B/(C+) of an extra storage array onto the cluster
<1> and it just puts all new files on the new one
<1> as long as the main file list knows which server id the files on, growth is pretty easy
<1> the A/B/C+ is redundant server copies
<1> jic situations
<3> what is this, like RAID in php? :-)
<1> so if server 2a falls over. as long as 2b,c, or d, etc are alive, things will just keep moving
<1> kinda :P
<0> Ukyo: i might need a setup like that, you have any references on that kind of setup
<1> you wouldnt belive how robust php can be
<0> ?
<1> i've been writing alot of daemons in php
<1> its just simple sql and file transfering
<1> file_id, file_name, fileserver_id
<3> Ukyo: why not jus setup RAID?
<1> each box has raid
<1> this is server level redundancy
<3> oh okay
<1> incase of kernel crashing etc
<1> similar to sql NDB clustering
<1> so
<1> imagine a 20TB system
<0> dools: you will always run into a limit on space... this design can grow to many TB's
<1> figure 5TB per server
<1> servers 1-5
<1> each box has its 5TB mirrord
<1> so technically 40TB of hdspace
<1> now, there is a 2nd set B servers
<1> of each of those 5
<1> so for 20TB storage array, your using 80TB of drive space
<1> But thats what happens when you want absolute redundancy. :)
<0> yep
<2> Hopefully the spares are off-site.
<1> actually
<1> once things get going
<1> there will be multiple connection sites
<0> Ukyo: you are saying this is a homegrown system?
<1> so customers will be able to choose their perfered connection
<1> they can do their data access from the LA, SFO, Dallas, DC, NYC servers
<3> Ukyo: so what if the server with the filesystem database on it crashes?
<1> fdaster connectivity
<1> dools: thats what NDB is for
<1> :)
<0> it almost sound like google but for them TB is like MBs
<1> you have a mysql cluster
<1> zircu: yeah, home based system
<2> Google doesn't believe in RAID.
<1> sure they do
<2> But I also was thinking about the similarities.
<1> they just do it at server level
<2> But, RAID implies disks :)
<0> Ukyo: i've been wanting to go to NDB, i have fulltext indexes that are very reliant on the system that has been stopping me
<1> ah
<1> honestly
<1> i havent setup NDB yet
<1> been eyeing the mysql setup docs
<1> on how to do it
<1> re-read it many times
<1> but i havent found anyone who can tell me hwo reliable it is
<0> ditto
<1> or speed, etc
<3> Ukyo: so you have several servers running the same scripts, and when they access a file, it goes through the mysql database to get at it, what about require'ing files?
<4> What's better practice: many arrays, or one array with information nested into dimensions?
<1> dools: re-phrase your question
<0> every time i read the docs i get diferent results...
<1> I like nests. They hold eggs. :P
<3> Ukyo: no
<2> Capso: I usually prefer the latter.
<4> Cwhiz: That means a bunch of nested foreach loops when utilising the array.
<0> Capso: well it depends on the struture
<0> and the purpose
<3> Capso: use nested arrays to group data together, so if you have a bunch of authors and each author has a bunch of books, then you would nest. if you have an array of fruit and an array of legumes, you wouldn't nest them
<4> Running multiple (like, two) SQL queries depending upon some variable... and putting the info. into two different arrays or one array...
<2> Capso: Do you know how to use joins?
<1> dools: user logs into ftp. requets blah.zip. server looks at its local read-only copy of the file listing table. check to see what fileserver id the file resides on. asks the local cache db if the file id is already locally stored, if not it adds a requsets for it. then connects to "a" fileserver with that file, and begins to stream to the user
<4> If one array... then it nests to two main dimensions, and one second dimension for one of them and for the other, up to three dimensions.
<2> Ukyo: Wait, so the app server and the cache server are different?
<1> yes
<4> Cwhiz: Yes, but in my case that's only helpful in ONE case.
<1> your saying 'app'
<2> Ukyo: Ohhhh....that makes sense now :)
<1> the whole thing is a big application
<1> technicalyl
<1> were talking file servers
<1> and data access servers
<1> :P
<3> Ukyo: okay, i think i get it, bt it's still a bit of a mystery to me as to what happens if the controlling server has a kernel panic
<3> s/bt/but
<2> It shouldn't be hard to dupe the control server.
<1> doolsL what are you calling the controlling server?
<2> The server with the database that points to the file servers.
<1> there is no centralised control here
<3> Ukyo: the ftp server that the user logged into in order to make the file request
<1> wqait wait
<1> look at it this way
<1> FTP server
<1> it has a clone DB
<5> Hi. Does anyone knows of an article or a tutorial about "good techniques" of handling $_POSTs and $_GETs in sql querys?
<1> that it keeps updates from the main NDB cluster
<3> mm2000: look at ww.php.net/mysql_real_escape_string
<2> mm2000: Google "SQL injection" PHP
<1> so it has a copy of the whole file list locally
<1> so it knows what fileserver id, to find the needed file on
<1> as well as a list of file clusters, and the ip's of each server and their backup server ip's
<1> there will be multiple multiple ftp servers
<1> they are all pretty stand-alone
<1> if one goes to connect to file server 4a, and its dead, it just hits 4b or 4c
<1> worst case scenario, it hits clusterB-4a
<1> etc
<3> Ukyo: so i am a user right, and i type in: ftp user@ftp.example.com and that dns resolves to ip address 223.45.68.9, and that server is dead, what happens?
<1> it wouldnt
<1> you woudlnt do it that way
<1> thats what load balancers are for
<1> :)
<1> if it returns 2 IP's
<1> both of those IP's are 2 diff load balancers
<1> the load balancer will have a list of IP's of ftp servers
<1> if an ftp server drops
<1> it removes its ip from the list until it responds correctly
<1> next question. :)
<3> okay, and what if the load balancing machine dies?
<1> **** happens. :)
<1> prolly have an extremly low ttl on the dns a record
<1> like 1 or 5 minutes
<1> and when they re-request, it will get the other ip's
<3> so this is my point: the kernel on the load balancing machine is just as likely to panic as any other, you are still relying on a single machine not crashing
<1> your always relying on that
<1> even yahoo
<1> nad google
<1> if you dns yahoo, and google
<1> those few ip's you get
Return to
#php
or
Go to some related
logs:
ubuntu php5 could not start up
#python
explain *** target pattern contains no `%'. Stop.
pastepin
#kde
opensuse j2ee cannot execute binary file
run msn messenger 7 on ubuntu wine
shut down a hard drive ubuntu
#linux
#mysql
|
|