forums.ps2dev.org Forum Index forums.ps2dev.org
Homebrew PS2, PSP & PS3 Development Discussions
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Barrier implementation using SPE signals

 
Post new topic   Reply to topic    forums.ps2dev.org Forum Index -> PS3 Development
View previous topic :: View next topic  
Author Message
ouasse



Joined: 30 Jul 2007
Posts: 90
Location: Paris, France
PostPosted: Thu Nov 20, 2008 4:18 am    Post subject: Barrier implementation using SPE signals Reply with quote
Hello,

As I've had little support about this problem from the IBM forum, maybe somebody here may be able to help me.

I want to implement an barrier between the SPE's, with no PPE intervention.

A barrier is a synchronization object which can be used to create meeting points between different processors. A processor which has called the barrier remains blocked until all other processors have also called the barrier.

My approach uses signals in ORed mode. My code looks okay to me, but unfortunately it hangs on my PS3. The program runs as expected on the Cell simulator (SDK 3.0).

The test code follows. first, a global definition (on both PPE and SPE) :

Code:
typedef struct {
  int spe_rank;
  int spe_count;
  int __dummy[2];
  uint64_t sig1[6];
} spe_data;



Then, the PPE program (includes removed):

Code:
#define SPE_COUNT 6
 
extern spe_program_handle_t barrier_sig_spu_handle;
 
typedef struct {
  spe_data d __attribute((aligned(16)));
  spe_context_ptr_t context;
  pthread_t thread;
} spe_id;
 
 
void *main_thr(void *ptr) {
  spe_id *id = (spe_id*)ptr;
  unsigned int entry_point = SPE_DEFAULT_ENTRY;
  int retval;
  do
    retval = spe_context_run(id->context, &entry_point, 0, &id->d, NULL, NULL);
  while (retval > 0);                  /* Run until exit or error */
  if(retval)
    perror("An error occurred running the SPE program");
  return NULL;
}
 
 
 
int main(int argc, char *argv[])
{
  spe_id spe[SPE_COUNT] __attribute((aligned(16)));
  int i, j;
  uint32_t dummy;
  uint64_t sig1_ea;
 
  for (i=0; i<SPE_COUNT; i++) {
    spe[i].d.spe_rank = i;
    spe[i].d.spe_count = SPE_COUNT;
    spe[i].context = spe_context_create(SPE_EVENTS_ENABLE
        | SPE_CFG_SIGNOTIFY1_OR | SPE_MAP_PS, NULL);
    spe_program_load(spe[i].context, &barrier_sig_spu_handle);
    sig1_ea = (unsigned int)spe_ps_area_get(spe[i].context,
        SPE_SIG_NOTIFY_1_AREA) + 12;
    for (j=0; j<SPE_COUNT; j++)
      spe[j].d.sig1[i] = sig1_ea;
  }
 
  for (i=0; i<SPE_COUNT; i++) {
    pthread_create(&spe[i].thread, NULL, main_thr, &spe[i]);
  }
 
  for (i=0; i<SPE_COUNT; i++) {
    while (!spe_out_mbox_status(spe[i].context));
    spe_out_mbox_read(spe[i].context, &dummy, 1);
  }
  dummy = 0;
  for (i=0; i<SPE_COUNT; i++)
    spe_in_mbox_write(spe[i].context, &dummy, 1, SPE_MBOX_ALL_BLOCKING);
 
  for (i=0; i<SPE_COUNT; i++) {
    pthread_join(spe[i].thread, NULL);
  }
  return 0;
}




And finally, the SPE code:

Code:
spe_data d __attribute((aligned(16)));
 
void spe_barrier(void) {
  volatile vec_uint4 signal;
  int i;
  void *ls = ((char*)&signal)+12;
  uint32_t expected = (1<<d.spe_count)-1;
  uint32_t received = 1<<d.spe_rank;
  signal = spu_promote(received, 3);
  for (i=0; i<d.spe_count; i++)
    if (i != d.spe_rank) {
      mfc_sndsig(ls, d.sig1[i], 4, 0, 0);
    }
  while (received != expected) {
    received |= spu_read_signal1();
    /*printf("spe%d received = %d\n", d.spe_rank, received);*/
  }
}
 
int main(unsigned long long spe_id, unsigned long long pdata)
{
  mfc_get(&d, pdata, sizeof(d), 0, 0, 0);
  mfc_write_tag_mask(1<<0);
  spu_mfcstat(MFC_TAG_UPDATE_ALL);
 
  spu_write_out_mbox(0);
  spu_read_in_mbox();
 
  printf("spe%d/%d: ready\n", d.spe_rank, d.spe_count);
  spe_barrier();
  printf("spe%d passed the barrier\n", d.spe_rank);
 
  return 0;
}


The basic idea about the algorithm is that every SPE waits for all the other SPE's to have reached the barrier. For this, the signal 1 channel has been configured in OR mode, so what a SPE reads in its signal 1 channel is a bit field corresponding to all the SPE's which already have reached the barrier. Then it loops waiting for all remaining SPE's to have reached the barrier, and returns.

On PS3, there are several cases:
- If the SPE code was compiled using -O2, only one SPE passes the barrier, all others hang
- If the SPE code was compiled without optimization, one, two or three SPE's pass the barrier
- If I comment out the printf in the spe_barrier function, the code works whatever the optimization level.

On the Cell simulator, every case works.

This optimization / printf difference makes me think about a synchronization problem, but my lack of experience does really not help me to find out the source of the problem.

I really wonder why a SPE can eventually not receive all the signals which have been sent to it.

What do you people think about this ? could it be a linux kernel problem ?

thank you in advance,

François
Back to top
View user's profile Send private message
IronPeter



Joined: 06 Aug 2007
Posts: 207
PostPosted: Fri Nov 21, 2008 12:27 am    Post subject: Reply with quote
Hmm... Probably you need the following code

while( !spu_stat_signal1( ) );
spu_read_signal1( );
Back to top
View user's profile Send private message Send e-mail Visit poster's website
ps2devman



Joined: 09 Oct 2006
Posts: 265
PostPosted: Fri Nov 21, 2008 12:36 am    Post subject: Reply with quote
Just for additional useless info..., MS and Nvidia call this 'barrier' mechanism a 'fence'... (may help to google and find more info about it)
Back to top
View user's profile Send private message
ouasse



Joined: 30 Jul 2007
Posts: 90
Location: Paris, France
PostPosted: Fri Nov 21, 2008 3:09 pm    Post subject: Reply with quote
IronPeter wrote:
Hmm... Probably you need the following code

while( !spu_stat_signal1( ) );
spu_read_signal1( );

There should be no reason why we would have to do it this way.

when calling spu_read_signal1(), the SPU is supposed to stall until a signal notification has been received. I tried your solution however, and it doesn't work.

Anyway, I really doubt about the integrity of my PS3 kernel. This program definitely should be working on the ps3, just like how it works on the Cell simulator. Could some of you try tris program on your ps3s, and tell me wheter it terminates correctly ? if it works, could you please tell me which kernel you are using (version, and kernel config file) ?

thank you very much.
Back to top
View user's profile Send private message
ouasse



Joined: 30 Jul 2007
Posts: 90
Location: Paris, France
PostPosted: Sat Nov 22, 2008 3:05 am    Post subject: Reply with quote
I have found a working solution. Putting
Code:
  mfc_write_tag_mask(1<<4);
  spu_mfcstat(MFC_TAG_UPDATE_ALL);

after the mfc_sndsig loop fixed the problem.

This still is quite strange, that I had to do that to fix the problem. The CBE handbook states that there is no need to wait for sndsig DMA commands to complete, as they mandatorily will.
Back to top
View user's profile Send private message
ouasse



Joined: 30 Jul 2007
Posts: 90
Location: Paris, France
PostPosted: Thu Nov 27, 2008 2:35 am    Post subject: Reply with quote
shame on me. The previous post was a workaround, not a solution.
The definitive solution was to declare the 'signal' variable as static in spe_barrier()
Back to top
View user's profile Send private message
IronPeter



Joined: 06 Aug 2007
Posts: 207
PostPosted: Fri Nov 28, 2008 1:13 am    Post subject: Reply with quote
Cool :). DMA from stack.

Yes, nice error.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    forums.ps2dev.org Forum Index -> PS3 Development All times are GMT + 10 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group