| |
| |
| |
|
Page: 1 2 3 4
Comments:
<0> geist: I was looking at your tmap stuff, really nice
<0> I'm kinda curious how you say... allocate half of virtual memory to some random allocator that chunks it up itself though
<0> do you call the allocator and say "40 megs for x" "60 megs for y" "all the rest for z" or something like that?
<0> err... rather call the tmap stuff
<0> or is there just no protection in the tmap stuff and it handles page transactions and sending back ranges of virtual memory
<1> the tmap stuff is just the very bottom of the vm
<1> the upper part of the vm is the bulk of the code, but it's arch-independent
<1> the tmap merely maps/unmaps/queries pages, at the request of the high level
<1> the high level allocates physical pages
<1> and carves up virtual space into regions of memory
<1> a region is a base + len of virtual memory with the same attributes
<1> ie, a mempory mapped file, or a chunk of zero backed memory
<1> the high level code is the one that takes page faults and attempts to service them
<1> it also tracks which physical pages are mapped where
<0> alright
<1> it's implemented in http://newos.org/WebSVN/listing.php?repname=NewOS&path=%2Fnewos%2Fkernel%2Fvm%2F&rev=0&sc=0
<0> my OS right now is userspace, then the kernel binary in memory, and then the space owned by the buddy allocator which allocates virtually contiguous stuff... in the future I'll pop in a DMA zone which uses wired pages, is there anything I'm forgetting?
<1> the buddy allocator allocates physical memory?
<0> nope, there's something like tmap
<1> ?
<0> just 3 linked lists of taken physical pages, used physical pages, and clean physical pages
<1> ah okay. mind you that has *nothing* to do with my tmap
<1> again, in newos, the translation map *only* deals with the details of using the mmu
<1> mapping pages
<1> it does not allocate, it doesn't track pages
<0> ohhh... so there's absolutely no tracking done whatsoever
<0> okay
<1> exactly
<1> the tracking is done somewhere else. basically like yours. a queue of page structures
<1> well, multiple queues
<0> oh... but it does have to track allocations for it's own page tables, right?
<1> no
<1> they're implicitly tracked by the fact that they're linked in the page directory
<1> when the tmap is freed, it simply iterates over the entries, freeing the page tables as well
<1> it could just as easily stick the vm_page entries in a list though
<1> in fact i might do that, to make the teardown slightly simpler
<0> oh, so each vm_page would be an action to take on the system's page tables during the free/commit?
<1> umm, not sure i understand that
<0> err... never mind, I think I got it
<1> it's probably simpler than you think
<1> the physical page allocator is it's own api
<1> you basically say something like vm_page *page = vm_allocate_phys_page();
<1> and the you own the structure, and are free to do whatever you want with that physical page
<1> the upper vm may use that to fulfill page faults, in which case it sticks the vm_page in a queue ***ociated with that vm object
<1> but the tmap code may also (for x86 at least) need a page to create a new page table
<1> so it uses the same api to allocate one on demand
<1> as long as you remember to put it back in the queue whent he tmap is free, it's all good
<1> as far as accounting, that page isnt' really ***ociated with any real vm object. it is just counted as overhead of the system
<0> hmm... okay. Do you keep the page tables themselves mapped in virtual memory ever?
<1> not permanently, no
<1> the page directory i do, but there's no really good reason to do that
<1> in newos i solve the 'somehow get to an arbitrary physical page' problem by keeping a range of kernel memory reserved for on-demand mappings of physical pages
<0> I'm starting to think I should just let the virtual allocator leave a little one page hole where where I can map in and out page tables for editing
<1> basically you can say map_physical_page() and it returns a pointer
<1> and you hand it back when you're done
<0> ah... it's really starting to come together now
<1> the physical page mapper keeps an LRU of mapped pages, so in theory stuff that's needed frequently wont get swapped out
<0> how big is that space?
<1> it's up to the architecture, but on x86 i think i gave it 256MB of kernel space
<1> and i map in 4MB chunks
<1> note that on some architectures this may be unnecessary
<1> for example, on most 64-bit machines you can just map of all physical memory into the kernel, and thus you dont need the LRU bits
<1> or for some more embedded cpus, you only have a finite amount of physical memory
<1> or the cpu has special instructions for accessing physical memory, etc
<0> ah, alright. It's certainly starting to come together now
<1> the api does have the advantage that you can map physical memory > 4GB the exact same way
<1> the VM can track up to i think 44bits of physical memory
<1> (since it uses 32-bit page numbers, which is the physical address shifted by 12 bits)
<1> thouhg it's untested. i dont have a machine with > 4GB of ram :)
<0> why so much kernel space for map_physical_page() stuff though?
<1> so that it doesn't have to actyally have to change the mapping that muych
<1> remember, it keeps an LRU of pages that had been mapped. it leaves them there after the last ref count goes away
<1> having to change kernel space pages is relatively expensive on an SMP machine, since you have to do a TLB shoot down on all the cpus
<1> do you understand what I mean?
<1> basically the 256MB is divided into 64 'slots' of 4MB a piece
<1> when you ask for a physical page, it looks to see if a 4MB slice already covers it
<1> if so, it increments the ref count, adds it to the end of the LRU list and hands them a pointer
<1> when they free it, it's ref goes to zero but the range isn't unmapped
<1> next time someone asks for a page and it finds that it's not allready mapped, it swaps the mapping with the oldest unused 4MB region
<1> of course there's a potential for deadlock there. in theory enough threads and/or cpus may have all the 'slots' active
<1> so you have to be able to deal with failure from the api
<1> and loop back and try again
<0> why 4 megs? Aren't page tables 4k?
<1> seemed like a nice size
<1> you dont want a ton of these slots because searching them would become slow
<1> but you dont want the granularity too high
<1> also, with 4MB you can use x86 4MB pages
<0> with that much why not keep all page tables mapped?
<1> though i ended up not doing that for other reasons
<1> because all the page tables are scattered throughout memory
<1> physical memory
<1> so what you suggest?
<1> there are many solutions to this mind you, this is the most flexible solution I could think of and still be reasonably simple and scalable
<0> well... initially I was thinking why not keep a single page hole where some function would lock, commit whatever changes you wanted to the table, unlock, and then return, but that wouldn't be very nice for the TLB
<0> err... there's mapping the physical page in question in there somewhere
<1> okay. that's essentially what I'm talking about here, but it's a more generalized api
<1> and for efficiency reasons it uses 4MB chunks
<1> mind you there are other users of this api. for zeroing out a page for example
<1> or doing copy on write
<0> ohhhhh
<0> that would make sense
<1> for security reasons you have to copy the data to the new page *before* mapping it
<1> since another thread on another cpu could be reading from the page as you do the copy
<0> right. Well, I may just do the single page thing for now and write it in a really general way until I feel like doing an LRU
<1> yup
<1> there are other strategies for dealing with that. linux has a hybrid strategy that's pretty nasty but fast
<1> they just map the first 896MB of physical space into the kernel (kernel space is 1GB, so that leaves 128 for code/data)
<1> and then they favor using that for certain types of page allocations
<1> then when you ask for a pointer to a physical page, if it's in the first 896 they just return a direct pointer
<1> otherwise, they have a little region like mine to deal with memory < 896
<0> hmm. Eh, your way is cooler.
<1> er >
<1> they have the advantage of being a lot faster for a lot of cases, with the added complexity of having to deal with multiple *types* of physical page allocation
<1> LOMEM, HIMEM, etc
<1> on some cpus this isna't an issue
<1> 64-bit ones, no biggie
<1> on 32bit ppc you'd put kernel space in it's own 4GB address space
<1> so you have a lot more space to reserve for these kind of things
<1> since swapping address spaces is essentially free
<1> actually it generally is for most other cpus other than x86. x86 is teh **** in the mmu department
<0> hmm... I know it's really early to start thinking about vesa, but once I get up to a good place I'd I'm thinking of doing a v86 monitor, but I'd like to do the video driver in kernel space... so what's the cleanest way to do that? I was thinking of linking in a 16 bit ***embly along with the driver page aligned and then I'd map that code segment to the first meg of virtual memory and schedule it.
<0> is there a much better way I'm not seeing?
<1> it's one of the reasons a lot of risc machines have traditionally run unixy systems much more efficiently, up until intel&amd just made cpus that are so much damn faster it doesn't matter
<1> no idea. i have absolutely zero interest in v86
<0> how'd you do it?
<1> never looked at it more than long enough to decide i never wanted to touch it
<1> do what? vesa? I dont
<1> i just use the bootloader to jam it into a mode and leave it
<1> and use a native driver for any serious graphics
<1> also basically 3 or 4 years ago i got a lot more interested in non PC systems, which mostly dont have these silly restrictions
<1> so i kind of pity the fools that keep hacking away at it :)
<0> yeah... You ever toy with SGI's boxes?
<0> they're fun
<1> yeah, i have a few
<1> got an indy around here somewhere
<1> only trouble is it is a r5000 based one, which apparently is a real PITA
<1> so i never got much past the bootloader
<0> sweet, I've got an octane that's a lot of fun to play with
<0> that and an O2... which is also apparently a pain to get to boot
<2> yawn, why do interesting discussions start up late?
<1> heh
<1> the most serious hacking attempt I made with newos was the dreamcast (hitachi SH) and a mac (PPC)
<1> actually got both of those ports all the way
<0> how hard is it to get the dreamcast to boot your kernel?
<1> pretty darn simple
<0> I've got one lying around
<1> folks immediately figured out how to burn CDs that boot directly
<1> and a buddy of mine wrote a simple UDP based loader
<0> yeah, I use mine for snes emulation and jet grind radio. I've booted linux on it a few times
<1> so it boots into the disc (takes about 10 seconds) and then there's a simple command line app that simply dumps your binary into memory over teh new and jumps into it
<1> and there's a serial port on the back of the dreamcast
Return to
#osdev
or
Go to some related
logs:
#linux
/dev/agpgart (AGP Support) genkernel
symbol lookup error: Xvnc: undefined symbol: cerr
wine wow xsession
#ubuntu
#mysql
awstats sendmail
#linux
cron job kdialog
iptables roundrobin
|
|