forums.ps2dev.org Forum Index forums.ps2dev.org
Homebrew PS2, PSP & PS3 Development Discussions
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Using compressed textures

 
Post new topic   Reply to topic    forums.ps2dev.org Forum Index -> PSP Development
View previous topic :: View next topic  
Author Message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Mon Jan 05, 2009 2:19 pm    Post subject: Using compressed textures Reply with quote

I've seen folks here who ran out of memory while loading images. It actually happened to me the other day, so I looked into compressed textures. In some cases, paletted textures may be the way to go. However, if you start with a program (porting a project is what I'm assuming here) that uses regular images, it would be a real pain to have to modify it to use paletted textures. I wanted something that could be done on the fly with a minimum of change to the original project.

That means compressed textures. The PSP supports DXT1, DXT3, and DXT5 compressed textures. I originally looked at a port of the squish library, but it's pretty slow and is a C++ library to boot. Libraries should always be C-based. It's easy to use a C lib in C++, but not the other way around. After looking through old threads, I found a GIMP plugin for exporting graphics for PSP projects on hitchhikr SoftWorks. It supported exporting DXT textures using libtxc_dxtn code. So I combined parts of that with parts of the "standard" graphics.c file people use quite a bit for loading pngs to make an example of how to load and compress an image, then use it.

The example uses the cube example as the base, with the png loading code from graphics.c, and the compressing from libtxc_dxtn.c. The new loadImage() function takes a second parameter - the texture mode to save the image as. It can be GU_PSM_8888, GU_PSM_DXT1, GU_PSM_DXT3, or GU_PSM_DXT5. If you specify GU_PSM_8888, it's much like the original code, except that the image is swizzled. From what I've seen, compressed textures can't be swizzled, so if you specify a DXT mode, it won't be swizzled, just compressed.

If you aren't familiar with the compressed modes, DXT1 is good for images without an alpha channel, and uses 1/8 the space of a 32 bit RGBA texture. DXT3 is good for textures with a sharp alpha, while DXT5 is good for general alpha images. DXT3 and DXT5 use 1/4 the space. So with DXT5, you could either use four times as many images, or you could use the same number of textures that are twice the height and width. Either case may help some projects.

http://www.mediafire.com/download.php?mbhl91ztwlz
Back to top
View user's profile Send private message AIM Address
willow :--)



Joined: 13 Jan 2007
Posts: 126

PostPosted: Mon Jan 05, 2009 2:45 pm    Post subject: Reply with quote

Thanks !
Back to top
View user's profile Send private message
hibbyware



Joined: 28 Mar 2007
Posts: 78

PostPosted: Mon Jan 05, 2009 9:38 pm    Post subject: Reply with quote

I think I will play around with this for a while today,

Thanks J.F.
Back to top
View user's profile Send private message
Art



Joined: 09 Nov 2005
Posts: 647

PostPosted: Tue Jan 06, 2009 9:41 am    Post subject: Reply with quote

Speaking of compression... what do you decompress that with?
Winrar says it's damaged.
_________________
If not actually, then potentially.
Back to top
View user's profile Send private message
willow :--)



Joined: 13 Jan 2007
Posts: 126

PostPosted: Tue Jan 06, 2009 9:55 am    Post subject: Reply with quote

Art wrote:
Speaking of compression... what do you decompress that with?
Winrar says it's damaged.

I had no problem with 7zip
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Tue Jan 06, 2009 10:17 am    Post subject: Reply with quote

Art wrote:
Speaking of compression... what do you decompress that with?
Winrar says it's damaged.


That's compressed with the standard command line version of 7zip. Worker (a file manager in linux) uses that to compress/decompress/list *.7z files automatically. I've gotten in the habit of using 7zip instead of zip as the files tend to be much smaller. Plus, many people know about 7z these days, so it's not quite as "mysterious" to folks as even a year ago. :)
Back to top
View user's profile Send private message AIM Address
Torch



Joined: 28 May 2008
Posts: 842

PostPosted: Tue Jan 06, 2009 2:28 pm    Post subject: Reply with quote

Thanks a lot. Now my app can be compatible with Phat PSP in XMB mode.
Back to top
View user's profile Send private message
Art



Joined: 09 Nov 2005
Posts: 647

PostPosted: Tue Jan 06, 2009 7:37 pm    Post subject: Reply with quote

Any chance of someone re hosting it zipped?
It's gonna take me 27 mins to dl the app to unzip it.

I assume this can be of assistance where an image takes twice the
RAM with Loadimagememory in graphics.c because it's stored both
as an array and image?
_________________
If not actually, then potentially.
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Tue Jan 06, 2009 8:29 pm    Post subject: Reply with quote

Art wrote:
Any chance of someone re hosting it zipped?
It's gonna take me 27 mins to dl the app to unzip it.


http://www.mediafire.com/download.php?ntt44znyntt


Quote:
I assume this can be of assistance where an image takes twice the
RAM with Loadimagememory in graphics.c because it's stored both
as an array and image?


? Not sure what you mean, but this example returns an image structure basically the same as the one you get from graphics.c, except that the data can be in one of four formats: RGBA (32 bit) just like "normal"; DXT1 which occupies 1/8 the space of the RGBA format; DXT3 which occupies 1/4 the space; or DXT5 which also occupies 1/4 the space. So if the image is 128x128, it used to occupy 128*128*4 + sizeof(struct Image). If you told it to be loaded as DXT3 or DXT5, it would occupy 128*128 + sizeof(struct Image). If you loaded it as DXT1, it would take 128*128/2 + sizeof(struct Image).

EDIT: Did you mean where the image was embedded into the app as data? That would be a case where it would take png-compressed-size + uncompressed-image-size amount of space. If that is what you mean, it would be png-compressed-size + dxt-compressed-size amount of space. The example loads the png from the memstick, and that's really the way you should do it if you're worried about saving ram in the PSP.
Back to top
View user's profile Send private message AIM Address
Art



Joined: 09 Nov 2005
Posts: 647

PostPosted: Tue Jan 06, 2009 9:27 pm    Post subject: Reply with quote

Thanks :)

Quote:

EDIT: Did you mean where the image was embedded into the app as data? That would be a case where it would take png-compressed-size + uncompressed-image-size amount of space. If that is what you mean, it would be png-compressed-size + dxt-compressed-size amount of space. The example loads the png from the memstick, and that's really the way you should do it if you're worried about saving ram in the PSP.\

Yes that's what I meant.
Yes I know loading direct from ms would save RAM, but where possible
I like single eboot programs.. seem a bit neater.
_________________
If not actually, then potentially.
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Tue Jan 06, 2009 10:34 pm    Post subject: Reply with quote

Certainly easier to install, but it's harder to update. Even if a single byte changes, you'd have to (potentially) redownload and copy a several MB file. If you keep the data files separate, only the file(s) that change need to be updated. This is particularly handy with games with lots of data files (or one BIG data file).

Also, like I mentioned, if you're REALLY tight on memory, you'll want to avoid embedded data if at all possible. However, this compression method might make it where you COULD keep the data embedded and still run in the space left over. So using compressed textures might be your only hope of making something with all the data embedded. So if that's what you really want, I can see why you'd be interested in this. :D
Back to top
View user's profile Send private message AIM Address
OldPrisoneR



Joined: 20 Mar 2008
Posts: 53

PostPosted: Thu Jan 08, 2009 12:28 am    Post subject: Reply with quote

Hi J.F., have you ever worked with texture matrix? e.g. make some texture rotation or translating
If so, can you post some examples?
Back to top
View user's profile Send private message
Torch



Joined: 28 May 2008
Posts: 842

PostPosted: Thu Jan 08, 2009 2:24 am    Post subject: Reply with quote

This would be useful when you want better resolution textures on the Slim. Your source files on the memory stick can be PNG lossless, and you can load it as 8888 on the Slim, or DXT on the Phat.

Its would also be useful when you want to dynamically load with different image quality in realtime depending on the current free memory.

However if you don't plan on using lossless source files, you might as well directly save your source files as DXT, because double compression would look really crappy.
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Thu Jan 08, 2009 9:16 am    Post subject: Reply with quote

Torch wrote:
This would be useful when you want better resolution textures on the Slim. Your source files on the memory stick can be PNG lossless, and you can load it as 8888 on the Slim, or DXT on the Phat.

Its would also be useful when you want to dynamically load with different image quality in realtime depending on the current free memory.

However if you don't plan on using lossless source files, you might as well directly save your source files as DXT, because double compression would look really crappy.


The main thing I was concerned about were ports that used a little too much memory to run on the PSP. Rather than require someone to compress a bunch of images (possibly thousands for some games) ahead of time, just compress on the fly. This makes it easier to deal with images in odd formats or in custom archives as well. You're compressing the image after it's loaded. However, it's up to the programmer to put it to good use. Many of the uses you describe could be done. As to quality, if you run the example I posted, you don't see much difference between 8888 mode and DXT mode for the png included.

OldPrisoneR wrote:

Hi J.F., have you ever worked with texture matrix? e.g. make some texture rotation or translating
If so, can you post some examples?


I'll see if I can do that... I've got some textures ripped from Tomb Raider that put several textures into a single image. The main thing to remember about DXT compression is that it uses a 4x4 block as the basic unit, so subtextures should fall on 4 pixel boundaries.
Back to top
View user's profile Send private message AIM Address
PosX100



Joined: 15 Aug 2007
Posts: 98

PostPosted: Thu Jan 08, 2009 7:01 pm    Post subject: Reply with quote

Nice stuff , thanks!.
Back to top
View user's profile Send private message
Kreationz



Joined: 18 May 2008
Posts: 53

PostPosted: Wed Sep 30, 2009 10:49 am    Post subject: Thanks, this should help Reply with quote

Well Daedalus is running out of Texel memory so this seems to be the way to go. Since we using complex texture formats. I guess the best method would be DXT5. By the way I've set this up as a Visual Studio Express solution for anyone that want to play with it.

@J.F.
Any chance you plan on doing more work on Daedalus?


Note: Here's a link to the project I made. No code has been changed just set it up for ease of use by us Windoze users

http://www.mediafire.com/download.php?jyxzznlyna4
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Thu Oct 01, 2009 3:00 am    Post subject: Reply with quote

Yeah, I've just been taking a short break to work on SEGA consoles for a while. Much of the work that is being done on DX64 is in an area that didn't concern me anyway (rendering support and optimization).
Back to top
View user's profile Send private message AIM Address
Kreationz



Joined: 18 May 2008
Posts: 53

PostPosted: Thu Oct 01, 2009 6:07 am    Post subject: Reply with quote

Well you already know this texture compression code so you can likely implement it faster than I could. (Some games need it.) Also, I have howard0su's ME related code if you wan to play with it with the -n64 branch.

Edit: Also with texture compression we can store the textures for future use longer and add to that converting 32bit to 16Bit and we should be able to have about 3X as much in our buffer.
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Fri Oct 02, 2009 2:28 am    Post subject: Reply with quote

Where in the code do you make the textures? I'll take a look to see about making compressed textures.
Back to top
View user's profile Send private message AIM Address
Kreationz



Joined: 18 May 2008
Posts: 53

PostPosted: Fri Oct 02, 2009 1:58 pm    Post subject: Reply with quote

That's where it get's complicated as it not like just loading a png due to the different endian they hove to be loaded convert then pushed to the texel memory. Not sure where they are loaded, but they are converted in ImageConvert.cpp (Trying to find the issue with CI4 textures leaving spots right now... likely the palette conversion.) But after they are loaded and coverted the texels are generated via GenerateTexels() in Texture.cpp the space for them gets allocated there as well. (64K downsize limit is a hack that cuases some thrashing, but otherwise we run out of memory fast on some games. Hence the need for compression.)
Back to top
View user's profile Send private message
J.F.



Joined: 22 Feb 2004
Posts: 2906

PostPosted: Sat Oct 03, 2009 2:22 am    Post subject: Reply with quote

Kreationz wrote:
That's where it get's complicated as it not like just loading a png due to the different endian they hove to be loaded convert then pushed to the texel memory. Not sure where they are loaded, but they are converted in ImageConvert.cpp (Trying to find the issue with CI4 textures leaving spots right now... likely the palette conversion.) But after they are loaded and coverted the texels are generated via GenerateTexels() in Texture.cpp the space for them gets allocated there as well. (64K downsize limit is a hack that cuases some thrashing, but otherwise we run out of memory fast on some games. Hence the need for compression.)


Where the PSP texture is MADE is where we want to be, so GenerateTexels() sounds like the function to alter. :)
Back to top
View user's profile Send private message AIM Address
Display posts from previous:   
Post new topic   Reply to topic    forums.ps2dev.org Forum Index -> PSP Development All times are GMT + 10 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group