forums.ps2dev.org Forum Index forums.ps2dev.org
Homebrew PS2, PSP & PS3 Development Discussions
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

PS3SDK collaboration invitation
Goto page 1, 2  Next
 
Post new topic   Reply to topic    forums.ps2dev.org Forum Index -> PS3 Development
View previous topic :: View next topic  
Author Message
mc



Joined: 12 Jan 2005
Posts: 212
Location: Linköping

PostPosted: Sat Jul 04, 2009 3:17 am    Post subject: PS3SDK collaboration invitation Reply with quote

Well, we've all heard the rumors about ongoing PS3SDK development efforts,
but so far nobody has shown any code.

I think the main problem with getting a PS3SDK out the door is that
it's a rather large project for one or two persons to undertake, so eventually
anyone who tries on his own will get stuck and/or fed up.

So what I'd like to do is invite to a collaborative effort, and the correct way to
do this is of course to present all the code I have so far, and allow people
to start expanding it.

What I have currently is a small multithreading kernel (supports use of both
HW threads), integration with newlib, and some simple GPU support.

There are plenty of work packages for interrested parties to get down with,
including (but not limited to)

* Interrupt handling
* A more sophisticated MMU driver
* Networking
* USB (the driver in mcload can't be used as it contains GPL code)
* Storage drivers and filesystems
* SPE support
* VMX support (currently context switching does not preserve vector registers)

To get going, check out the following bzr repository:

http://mc.pp.se/bzr/ps3sdk

Note that you will need to rebuild newlib with the patch contained in
the "misc" directory (simply copy the patch over the old one in the ps3toolchain
distribution).

If you want to start working on something, please let me know, and preferably
publish the URL of your own bzr branch, so that others can follow your progress,
and so that I can merge things back to my main branch.

Let's get hacking, people!
_________________
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
Back to top
View user's profile Send private message
jbit
Site Admin


Joined: 28 May 2005
Posts: 293
Location: København, Danmark

PostPosted: Sat Jul 04, 2009 3:26 am    Post subject: Reply with quote

It would be nice if we can keep the core GPL clean, as in all code is BSD/AFPL/whatever licensed, which means of course no copying code/headers/etc from linux..
GPL causes a lot of headaches at later stages for a lot of people, so it's best to avoid for this kind of thing.

(Other peoples opinions are welcome of course :P)
Back to top
View user's profile Send private message Visit poster's website
mc



Joined: 12 Jan 2005
Posts: 212
Location: Linköping

PostPosted: Sat Jul 04, 2009 3:37 am    Post subject: Reply with quote

@jbit: Yes, that's why I mentioned this issue with the existing USB driver.

I've reused the LICENSE from the PSPSDK.

Currently I have lv1callarg.h, lib1calltab.h and hv.S still copied from Linux for generating
the HV call stubs, so I guess another work package would be to cleanroom that.
_________________
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
Back to top
View user's profile Send private message
jbit
Site Admin


Joined: 28 May 2005
Posts: 293
Location: København, Danmark

PostPosted: Sat Jul 04, 2009 3:43 am    Post subject: Reply with quote

Ah, I missed that sorry... But even so it makes sense to make people aware :)

Also I think we have enough info on the wiki about the lv1 interface to validly make a cleanroom BSD implementation. Should only be a days work or so.
Back to top
View user's profile Send private message Visit poster's website
mc



Joined: 12 Jan 2005
Posts: 212
Location: Linköping

PostPosted: Sat Jul 04, 2009 3:51 am    Post subject: Reply with quote

Yup, then information in the wiki is more than adequate (you actually only need
<name, id, # of inputs, # of outputs> for each call). I've already started creating
a header file with prototypes for the HV calls based on the wiki (src/hypervisor/ps3hv.h).

Hm, one file I'm a little unsure about is the linkfile, which is based on the linker
script generated by binutils. It does not contain any license information,
so I don't know if it is GPL, LGPL, or something different.
_________________
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Sat Jul 04, 2009 7:21 am    Post subject: Reply with quote

It would also be helpful if the different parts (like usb or net) were separate and independent as much as possible. For example, let's say you made a PC or Mac emulator and just wanted to pass the hardware for the USB on to the emulator. If you FORCE people to load a USB driver, then you also FORCE people to emulate the hardware using the driver in some cases, like the one just mentioned. Now clearly an input driver would require a USB driver, but if you don't use anything that needs it, don't force people to load/use it anyway.
Back to top
View user's profile Send private message AIM Address
mc



Joined: 12 Jan 2005
Posts: 212
Location: Linköping

PostPosted: Sat Jul 04, 2009 7:35 am    Post subject: Reply with quote

The approach so far is pretty modular. I don't even FORCE anyone to turn on the MMU,
although of course most drivers won't work without it... ;-)
_________________
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Sat Jul 04, 2009 9:32 am    Post subject: Reply with quote

mc wrote:
The approach so far is pretty modular. I don't even FORCE anyone to turn on the MMU,
although of course most drivers won't work without it... ;-)


Good. I've done a checkout and will look it over to see what I can contribute.
Back to top
View user's profile Send private message AIM Address
ps2devman



Joined: 09 Oct 2006
Posts: 265

PostPosted: Mon Jul 06, 2009 4:04 pm    Post subject: Reply with quote

Strong advise : implement right from the beginning a strong 'DPC' mechanism for all interrupts. If you get strange and impossible to understand issues think about interrupt that fires while you are already handling the previous one. The direct interrupt handling routing must allow reentrance and be as fast as possible. Don't treat interrupt there, just enqueue what you need to treat it later. Then have something that dequeues sequentially events and call clean DPC's (deferred procedure calls) to treat them. The DPC's are normal routines and can be interrupted anytime like any other code part.

(can someone write the url to some page that explain how bzr work?
for me page is empty...)
Back to top
View user's profile Send private message
mc



Joined: 12 Jan 2005
Posts: 212
Location: Linköping

PostPosted: Mon Jul 06, 2009 9:07 pm    Post subject: Reply with quote

Yes, I was also thinking about this kind of solution for interrupt processing. I have experience from work with commercial microkernels which use this method, and it really makes stuff easier for the clients. For it to work properly, I think preemption needs to be added to the scheduler though (it doesn't have it yet).

For a quick bzr tutorial, see http://doc.bazaar-vcs.org/latest/en/mini-tutorial/index.html. Basically, you just do "bzr branch htpp://mc.pp.se/bzr/ps3dev" to get your own branch that you can commit locally to.
_________________
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
Back to top
View user's profile Send private message
nbauernf



Joined: 18 Nov 2008
Posts: 13
Location: Carnegie Mellon University

PostPosted: Tue Jul 07, 2009 4:36 am    Post subject: Reply with quote

Hmm I'm having trouble checking the branch out.

$ bzr branch http://mc.pp.se/bzr/ps3sdk
bzr: ERROR: Invalid http response for http://mc.pp.se/bzr/ps3sdk/: Unable to handle http code 501: Not Implemented

Any clues?
Back to top
View user's profile Send private message
nbauernf



Joined: 18 Nov 2008
Posts: 13
Location: Carnegie Mellon University

PostPosted: Tue Jul 07, 2009 5:02 am    Post subject: Reply with quote

Turns out adding a '--no-plugins' took care of that problem!
Back to top
View user's profile Send private message
gkr



Joined: 09 Jul 2009
Posts: 1

PostPosted: Thu Jul 09, 2009 6:47 am    Post subject: Reply with quote

Hi,
I get error message when trying to check out

c:\Python24\Scripts>bzr branch http://mc.pp.se/bzr/ps3sdk
bzr: ERROR: exceptions.NotImplementedError: We must have one of fcntl, pywin32,
or ctypes available to support OS locking.
Back to top
View user's profile Send private message
mc



Joined: 12 Jan 2005
Posts: 212
Location: Linköping

PostPosted: Thu Jul 09, 2009 7:23 am    Post subject: Reply with quote

@gkr: Did you follow the instructions here?

Or maybe you are using MS-DOS?
_________________
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
Back to top
View user's profile Send private message
ps2devman



Joined: 09 Oct 2006
Posts: 265

PostPosted: Thu Jul 09, 2009 5:21 pm    Post subject: Reply with quote

If you install the windows version of bazaar with minimum installation
(no plugin) it definitely works without a fuss (at least on xp)

bazaar in 5 minutes :
http://doc.bazaar-vcs.org/bzr.dev/en/mini-tutorial/index.html

You don't even need bazaar in the path if you specify a path for bzr in a .bat file. Create a .bat file in a c:\bzr_projects directory with this inside :

c:\progra~1\bazaar\bzr whoami "myname <myemail>"
c:\progra~1\bazaar\bzr branch http://mc.pp.se/bzr/ps3sdk

and launch it
Back to top
View user's profile Send private message
rapso



Joined: 28 Mar 2005
Posts: 147

PostPosted: Mon Jul 13, 2009 10:20 pm    Post subject: Reply with quote

A homebrew PS3SDK would be definitely incredibily cool :),
but from the text here it rather seems to be another OS. is there something I miss? (that's no offense. I just don't really see the goal of this project as the topic seems to promise something else than the content, imho).


rapso
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Tue Jul 14, 2009 12:57 am    Post subject: Reply with quote

OTHEROS means the program is running on bare metal with only the hypervisor below it. There are no libraries or anything else to help do stuff, so yes, it should seem like another OS when discussing the SDK - it needs to do everything the OS would do.
Back to top
View user's profile Send private message AIM Address
rapso



Joined: 28 Mar 2005
Posts: 147

PostPosted: Tue Jul 14, 2009 7:07 am    Post subject: Reply with quote

so it's mcload with BSD license?

for me, it wouldn't need to do all the stuff to get into development. flat memory (instead of an sophisticated MMU driver), no context switching (just a call into main(..) and main2(...) with two HW threads), video out and pad would be all needed from an OS to get into it, similar to PS2 dev.

I don't say this is bad, I even praise you guys for doing such a community work, I just want to say I (and maybe other guys) would be happy about far less, as long as it's simple to get running and with access to basic hardware to get (homebrew)games running.
Back to top
View user's profile Send private message
emoon



Joined: 18 Jan 2004
Posts: 91
Location: Stockholm, Sweden

PostPosted: Tue Jul 14, 2009 7:48 pm    Post subject: Reply with quote

I will release the code I have worked on to mc.

It has a semi working Ethernet driver that can actually receive and start an elf (the code is very crude and needs lots of work though)

I don't have much time to work on it but hopefully it will give some base to work from at least.
Back to top
View user's profile Send private message Visit poster's website
emoon



Joined: 18 Jan 2004
Posts: 91
Location: Stockholm, Sweden

PostPosted: Tue Jul 14, 2009 8:34 pm    Post subject: Reply with quote

Ok, I actually forgot stuff I had implemented :)

* tcp/ip (semi working) and is using LWIP
* More cleaned up code
* echo server over tcp/ip (telenet style that echos back what you send it)
* lv1 calls renamed to lv1NetControl(...) instead of lv1_net_control(...) to try to remove the 'GPLisnes' of it and also some other changes to try to make it different from the linux code
* Totally custom written smb support (to be used to load files remote) It has been implemented by Chip and is way more lean than samba and focus on whats needed (never tested on ps3 afaik but code is clean and documented)
* Includes docs on the coding standard we intended to use
* Includes a early version of ps3link (its able to kick elfs, but not very tested and needs to be fixed up to be really useable)
* Also a early version of ps3client is included.

Hopefully you will find the work that I and Chip have done useful
Back to top
View user's profile Send private message Visit poster's website
mc



Joined: 12 Jan 2005
Posts: 212
Location: Linköping

PostPosted: Wed Jul 15, 2009 6:46 am    Post subject: Reply with quote

emoon wrote:
* lv1 calls renamed to lv1NetControl(...) instead of lv1_net_control(...) to try to remove the 'GPLisnes' of it

Actually, I'm not sure this is productive. You can't copyright an API, and I'm pretty sure these names come from the IBM/Sony HV code anyway, so I don't see any "GPLisnes" to remove as far as the actual names are concerned.

But if others have a different opinion, feel free to motivate. :)
_________________
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
Back to top
View user's profile Send private message
mc



Joined: 12 Jan 2005
Posts: 212
Location: Linköping

PostPosted: Wed Jul 15, 2009 6:58 am    Post subject: Reply with quote

@rapso:

If you don't want to use the MMU driver, that's fine. Just don't call ps3MmuInit () and ps3MmuOn(), and you'll get your flat memory model. Note however that due to the way hardware virtualization works in the Cell, the flat memory model contains 128MB of RAM _only_. No video memory, no USB register file, no nothing. So for things like the console driver and any upcoming USB driver, the fancy schmancy MMU stuff is _required_. It isn't there just for kicks.

Same with the scheduler; if you just create one new thread from main (your main2), and never call any functions which can deschedule (sleep, mutex operations etc), you will never get any context switches. But some divers may require them because it is impractical to make them otherwise,
_________________
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
Back to top
View user's profile Send private message
rapso



Joined: 28 Mar 2005
Posts: 147

PostPosted: Wed Jul 15, 2009 7:52 pm    Post subject: Reply with quote

thx for clarifying that MC, I wasn't aware of all those implications that makes all those tasks u listed a _must_ (I really miss the console feeling tho)
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Thu Jul 16, 2009 1:40 am    Post subject: Reply with quote

rapso wrote:
(I really miss the console feeling tho)


The problem is that modern "consoles" aren't consoles anymore - they're compact computers. Just as you wouldn't try to write a PC ethernet driver that ignored memory mapping, PS3 drivers shouldn't ignore memory mapping as well.
Back to top
View user's profile Send private message AIM Address
maaku



Joined: 31 Aug 2009
Posts: 4

PostPosted: Mon Aug 31, 2009 6:06 pm    Post subject: Contiki OS? Reply with quote

mc, this is an interesting project. I would like to help you out, although the realities of my other commitments are such that I cannot make any promises.

Might I suggest though that instead of writing an operating system from scratch (as much fun as that might be) we should instead focus on porting an appropriately small, but functional embedded OS like Contiki? That way we can focus on providing PS3 API's rather than repetitive low-level tasks...
Back to top
View user's profile Send private message
maaku



Joined: 31 Aug 2009
Posts: 4

PostPosted: Mon Aug 31, 2009 6:37 pm    Post subject: Reply with quote

Or MINIX, actually. That might be a better route to take than Contiki.
Back to top
View user's profile Send private message
rapso



Joined: 28 Mar 2005
Posts: 147

PostPosted: Mon Aug 31, 2009 7:27 pm    Post subject: Reply with quote

Why porting another OS if you can use linux?
I think the point was to have something lightweight from scratch dedicated for this one and only purpose: homebrew gamedev on ps3.
Back to top
View user's profile Send private message
maaku



Joined: 31 Aug 2009
Posts: 4

PostPosted: Tue Sep 01, 2009 2:33 am    Post subject: Reply with quote

Because when you look at what people really want for a homebrew development, Linux would not satisfy their needs. For homebrew you want an embedded operating system (if anything at all). Linux is a server OS. The two extremes couldn't be more different.

It would not be a very large undertaking. MINIX is already ported to the PowerPC, and the kernel itself is only 4000 lines of code.

EDIT: Another option (just to throw it out there) would be L4Ka::Pistachio.
Back to top
View user's profile Send private message
rapso



Joined: 28 Mar 2005
Posts: 147

PostPosted: Tue Sep 01, 2009 9:39 am    Post subject: Reply with quote

maaku wrote:
Because when you look at what people really want for a homebrew development, Linux would not satisfy their needs.

Thats simply not true. the linux kernel is very generic and exists in tons of versions,
all packages are optional, there are kinda rip-offs here at the forum that are already some basic game development os, the only issue ppl saw was the licensing via gpl, afaik.

Quote:

For homebrew you want an embedded operating system (if anything at all). Linux is a server OS. The two extremes couldn't be more different.
there are plently of embeded linux versions, from realtime to pure virtual machine aimed like android.

Quote:

It would not be a very large undertaking. MINIX is already ported to the PowerPC, and the kernel itself is only 4000 lines of code.
as far as i know, minix is just a micro kernel, so its probably not doing more than MC's current version of the gamedev os.

Quote:

EDIT: Another option (just to throw it out there) would be L4Ka::Pistachio.

i think, as long as the license is compatible with the bsd alike license this new game dev os is made for, you can rip of any other os and "port" it to this one. but lets see what MC says:)
Back to top
View user's profile Send private message
maaku



Joined: 31 Aug 2009
Posts: 4

PostPosted: Tue Sep 01, 2009 12:35 pm    Post subject: Reply with quote

rapso,

I'm sorry I don't have the time to engage in a full discussion on this, but let me offer at least a few points (from over a decade of experience in embedded software, linux kernel customization, and game development):

* Linux can be streamlined to an embedded profile, but at the cost of many man hours initially and continually in maintenance. Easy for a corporation to do, not so much for a loose-nit homebrew community.

* The existing kernels are all desktop/server derived variants of Linux. There is not, to my knowledge, any true real-time Linux kernel for the PS3/Cell. Real-time linux is always highly-coupled with the architecture that it is designed for, and developing such a kernel for PS3/Cell will take a non-trivial amount of time and resources to do properly. It's not just a matter of hitting some checkboxes and compiling.

* The ps3sdk has very small, but very specific needs that do not match up very closely with Linux (even with the patches Sony and those on this forum have made).

* Don't fall into the "Linux will be easier because it's Linux" fallacy. Linux is around ~3 million lines of code. MINIX is just 4000 lines of kernel code. L4 is of similar size, and already ported to 64-bit PowerPC. That's three orders of magnitude difference (yes, I know we are comparing micro- to macro-kernels, but for reasons of software engineering complexity it is relevant). What's more, these kernels already have reliable implementations of feature sets needed for real-time embedded programming and are simple enough to not require the extensive tweaking in the port that Linux RTOS options do.

Most people don't know, or underestimate even if they do the amount of work that must go into creating and maintaining a custom linux kernel for a real-time application. It is my professional opinion that the ps3sdk could be finished faster, and with better satisfaction by adopting an existing, smaller, non-Linux solution.

EDIT: But perhaps we should agree to disagree, if you're still not convinced. My free time is very limited, and any part of it I spend here would otherwise be spent on ps3sdk. And I think we all agree that flawed approach or not, any ps3sdk is better than no ps3sdk.


MC, are you still here? Unless I hear from you, I'm going to begin porting either MINIX or L4 to Cell, but I could use some of your expert knowledge... (or anyone else here that is familiar with kernel mode PPE).
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.ps2dev.org Forum Index -> PS3 Development All times are GMT + 10 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group