PostGIS (actually liblwgeom) integration with oss-fuzz

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

PostGIS (actually liblwgeom) integration with oss-fuzz

Even Rouault-2

Hi,

 

I've prototyped an integration of liblwgeom from PostGIS repository with oss-fuzz.

 

Quoting https://github.com/google/oss-fuzz/ ,

"""

Fuzz testing is a well-known technique for uncovering various kinds of programming errors in software.

Many of these detectable errors (e.g. buffer overflow) can have serious security implications.

 

We successfully deployed guided in-process fuzzing of Chrome components and found hundreds of

security vulnerabilities and stability bugs. We now want to share the experience and the

service with the open source community.

 

In cooperation with the Core Infrastructure Initiative, OSS-Fuzz aims to make common

open source software more secure and stable by combining modern fuzzing techniques

and scalable distributed execution.

""""

 

GDAL and proj.4 have joined oss-fuzz, for a few weeks and this is really efficient. I've fixed between 300 and 400 bugs in GDAL...

 

So I just gave it a try with PostGIS, concentrating on liblwgeom, since it builds nicely in oss-fuzz environment

(plain "make" in top repository fails in oss-fuzz from some reason I haven't investigated)

As an example, I've created 2 fuzzers, one for lwgeom_from_wkb() and the other one for lwgeom_from_wkt().

More could be done, based on those examples. Left as an exercice to other developers.

 

Integration of a software with oss-fuzz is made of 2 parts:

- fuzzer entry points must be in the project repository : https://github.com/rouault/postgis/tree/ossfuzz/fuzzers

- a metadata file (project.yaml), a Dockerfile (download needed packages & PostGIS source code) and

a "bootstrap" build.sh script must be integrated in OSS-Fuzz own repo too :

https://github.com/rouault/oss-fuzz/tree/postgis/projects/postgis

 

For now, I've done this in my own postgis and ossz-fuzz git forks as you can see, but ultimately if the

project agrees we should merge this into their respective official repos.

 

But pending that, you can already try this stuff locally

{{{

Make sure you have Docker installed

 

git clone --branch postgis https://github.com/rouault/oss-fuzz

cd oss-fuzz

 

Build the Docker image:

python infra/helper.py build_image postgis

 

Build PostGIS and the fuzzer programs with the address sanitizer

python infra/helper.py build_fuzzers --sanitizer address postgis

 

Run one of the fuzzer (you can try with wkt_import_fuzzer too)

python infra/helper.py run_fuzzer postgis wkb_import_fuzzer

}}}

 

My local experiments show that lwgeom_from_wkt() seems to be rather robust,

but lwgeom_from_wkb() has a few bugs. For the record, I found and fixed (in my fork) this memory leak

(memory leak = leak in a case where lw_error() is not called) in

https://github.com/rouault/postgis/commit/cf179396b719223653eee56a01189339e0abcc0d

There's also a heap buffer overflow it just detected in

ptarray_from_wkb_state /src/postgis/liblwgeom/lwin_wkb.c:367

 

So if the community is interested in a closer integration in OSS Fuzz, next steps are :

1) someone with PostGIS commit rights merges https://github.com/rouault/postgis/commit/0181a28ab01764b4e6d11a5d2ffe7edce96498c6

into PostGIS SVN (as well as the bug fix https://github.com/rouault/postgis/commit/cf179396b719223653eee56a01189339e0abcc0d while you are it)

2) interested core PostGIS developers give me a @gmail.com email, so I add it in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/project.yaml

This way they will then have access to the bug reports that are embargoed for

90 days (or 30 days after OSS Fuzz has found them to be fixed)

3) I then modify https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/Dockerfile to

point to PostGIS official github mirror instead of my fork

4) I then submit a pull request to https://github.com/google/oss-fuzz/ with my

https://github.com/rouault/oss-fuzz postgis branch . They may accept or not the application, but I guess they will accept.

If they don't, you can also play with it locally as I showed above. And this is strongly recommended

to do so when adding a new fuzzer for example.

5) once the project is accepted, monitor https://bugs.chromium.org/p/oss-fuzz/issues/list?q=postgis and fix the bugs !

6) add more fuzzers. Hint: in PostGIS "fuzzers" directory, "make dummyfuzzers" to check that your fuzzer builds.

 

Note: I don't volunteer to fix all bugs that will be found. I have already enough to do with GDAL... I wouldn't

mind if someone wants to be the declared maintainer in oss-fuzz projects/postgis/project.yaml and projects/postgis/Dockerfile

 

Note 2: if you look closely at https://github.com/rouault/postgis/blob/ossfuzz/fuzzers/wkb_import_fuzzer.cpp

you will notice that it is a bit messy since it stubs GEOS and geod_ symbols. This is due to the fact

that OSS-Fuzz requires that the fuzzer programs are completely statically linked, and Ubuntu doesn't ship

with static builds of geos (actually just libgeos.a but no libgeos_c.a) as far as I can see.

All this could be improved by adding a download of GEOS source code in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/Dockerfile and building it manually in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/build.sh

 

Thoughts ?

 

Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Paul Ramsey-3
This seems very cool but also very involved and complicated :) 
I think everyone is concomitantly afraid to step forward and pick it up. It's got it all: external services, big chains of dependencies, docker :) 
I think everyone is a little afraid to pick it up, lest they own it.
P.


On Sun, Jul 2, 2017 at 12:07 PM, Even Rouault <[hidden email]> wrote:

Hi,

 

I've prototyped an integration of liblwgeom from PostGIS repository with oss-fuzz.

 

Quoting https://github.com/google/oss-fuzz/ ,

"""

Fuzz testing is a well-known technique for uncovering various kinds of programming errors in software.

Many of these detectable errors (e.g. buffer overflow) can have serious security implications.

 

We successfully deployed guided in-process fuzzing of Chrome components and found hundreds of

security vulnerabilities and stability bugs. We now want to share the experience and the

service with the open source community.

 

In cooperation with the Core Infrastructure Initiative, OSS-Fuzz aims to make common

open source software more secure and stable by combining modern fuzzing techniques

and scalable distributed execution.

""""

 

GDAL and proj.4 have joined oss-fuzz, for a few weeks and this is really efficient. I've fixed between 300 and 400 bugs in GDAL...

 

So I just gave it a try with PostGIS, concentrating on liblwgeom, since it builds nicely in oss-fuzz environment

(plain "make" in top repository fails in oss-fuzz from some reason I haven't investigated)

As an example, I've created 2 fuzzers, one for lwgeom_from_wkb() and the other one for lwgeom_from_wkt().

More could be done, based on those examples. Left as an exercice to other developers.

 

Integration of a software with oss-fuzz is made of 2 parts:

- fuzzer entry points must be in the project repository : https://github.com/rouault/postgis/tree/ossfuzz/fuzzers

- a metadata file (project.yaml), a Dockerfile (download needed packages & PostGIS source code) and

a "bootstrap" build.sh script must be integrated in OSS-Fuzz own repo too :

https://github.com/rouault/oss-fuzz/tree/postgis/projects/postgis

 

For now, I've done this in my own postgis and ossz-fuzz git forks as you can see, but ultimately if the

project agrees we should merge this into their respective official repos.

 

But pending that, you can already try this stuff locally

{{{

Make sure you have Docker installed

 

git clone --branch postgis https://github.com/rouault/oss-fuzz

cd oss-fuzz

 

Build the Docker image:

python infra/helper.py build_image postgis

 

Build PostGIS and the fuzzer programs with the address sanitizer

python infra/helper.py build_fuzzers --sanitizer address postgis

 

Run one of the fuzzer (you can try with wkt_import_fuzzer too)

python infra/helper.py run_fuzzer postgis wkb_import_fuzzer

}}}

 

My local experiments show that lwgeom_from_wkt() seems to be rather robust,

but lwgeom_from_wkb() has a few bugs. For the record, I found and fixed (in my fork) this memory leak

(memory leak = leak in a case where lw_error() is not called) in

https://github.com/rouault/postgis/commit/cf179396b719223653eee56a01189339e0abcc0d

There's also a heap buffer overflow it just detected in

ptarray_from_wkb_state /src/postgis/liblwgeom/lwin_wkb.c:367

 

So if the community is interested in a closer integration in OSS Fuzz, next steps are :

1) someone with PostGIS commit rights merges https://github.com/rouault/postgis/commit/0181a28ab01764b4e6d11a5d2ffe7edce96498c6

into PostGIS SVN (as well as the bug fix https://github.com/rouault/postgis/commit/cf179396b719223653eee56a01189339e0abcc0d while you are it)

2) interested core PostGIS developers give me a @gmail.com email, so I add it in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/project.yaml

This way they will then have access to the bug reports that are embargoed for

90 days (or 30 days after OSS Fuzz has found them to be fixed)

3) I then modify https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/Dockerfile to

point to PostGIS official github mirror instead of my fork

4) I then submit a pull request to https://github.com/google/oss-fuzz/ with my

https://github.com/rouault/oss-fuzz postgis branch . They may accept or not the application, but I guess they will accept.

If they don't, you can also play with it locally as I showed above. And this is strongly recommended

to do so when adding a new fuzzer for example.

5) once the project is accepted, monitor https://bugs.chromium.org/p/oss-fuzz/issues/list?q=postgis and fix the bugs !

6) add more fuzzers. Hint: in PostGIS "fuzzers" directory, "make dummyfuzzers" to check that your fuzzer builds.

 

Note: I don't volunteer to fix all bugs that will be found. I have already enough to do with GDAL... I wouldn't

mind if someone wants to be the declared maintainer in oss-fuzz projects/postgis/project.yaml and projects/postgis/Dockerfile

 

Note 2: if you look closely at https://github.com/rouault/postgis/blob/ossfuzz/fuzzers/wkb_import_fuzzer.cpp

you will notice that it is a bit messy since it stubs GEOS and geod_ symbols. This is due to the fact

that OSS-Fuzz requires that the fuzzer programs are completely statically linked, and Ubuntu doesn't ship

with static builds of geos (actually just libgeos.a but no libgeos_c.a) as far as I can see.

All this could be improved by adding a download of GEOS source code in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/Dockerfile and building it manually in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/build.sh

 

Thoughts ?

 

Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Even Rouault-2

On mardi 4 juillet 2017 14:38:28 CEST Paul Ramsey wrote:

> This seems very cool but also very involved and complicated :)

 

Well, I did at least all the setup part :-) So basically just merging my commit is needed to have it set up and automatically find and file bugs (and auto-close the reports once the bug fix has been committed and verified the day following the commit)

 

Fixing bugs is the only part that isn't done (yet ?) automatically.

 

Maintainance of the system should be rather low, unless PostGIS build system and tree structure changes often, which doesn't seem to be the case.

 

I can give more explanations if needed. I initiated a README.TXT in

https://github.com/rouault/postgis/blob/ossfuzz/fuzzers/README.TXT

 

> I think everyone is concomitantly afraid to step forward and pick it up.

> It's got it all: external services, big chains of dependencies, docker :)

 

As I showed, you can also relatively easily play with it locally too.

 

> I think everyone is a little afraid to pick it up, lest they own it.

 

Yes, sure. But I guess that will become as common as using Travis-CI, AppVeyor or other continuous integration systems for OSS projects. Just look at the list of projects already integrated at

https://github.com/google/oss-fuzz/tree/master/projects

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Regina Obe-2
In reply to this post by Paul Ramsey-3

Okay Even sold me on this and convinced me it's not as much work as I'm expecting it to be.  I guess I'll step forward.

 

I'll submit a request to OSS fuzz hopefully this weekend for the continuous integration service and pull in Even's setup.

 

I'll also commit the memory leak fix that Even found and provided a patch for.

 

Even did you by chance see anything funny with Kmeans?  That cunit thing is driving me nuts cause it fails 25% of the time on windows (more on 32-bit runs)with some crash.

There's probably something amiss there.

 

Thanks,

Regina

 

From: postgis-devel [mailto:[hidden email]] On Behalf Of Paul Ramsey
Sent: Tuesday, July 04, 2017 5:38 PM
To: PostGIS Development Discussion <[hidden email]>
Subject: Re: [postgis-devel] PostGIS (actually liblwgeom) integration with oss-fuzz

 

This seems very cool but also very involved and complicated :) 

I think everyone is concomitantly afraid to step forward and pick it up. It's got it all: external services, big chains of dependencies, docker :) 
I think everyone is a little afraid to pick it up, lest they own it.

P.

 

 

On Sun, Jul 2, 2017 at 12:07 PM, Even Rouault <[hidden email]> wrote:

Hi,

 

I've prototyped an integration of liblwgeom from PostGIS repository with oss-fuzz.

 

Quoting https://github.com/google/oss-fuzz/ ,

"""

Fuzz testing is a well-known technique for uncovering various kinds of programming errors in software.

Many of these detectable errors (e.g. buffer overflow) can have serious security implications.

 

We successfully deployed guided in-process fuzzing of Chrome components and found hundreds of

security vulnerabilities and stability bugs. We now want to share the experience and the

service with the open source community.

 

In cooperation with the Core Infrastructure Initiative, OSS-Fuzz aims to make common

open source software more secure and stable by combining modern fuzzing techniques

and scalable distributed execution.

""""

 

GDAL and proj.4 have joined oss-fuzz, for a few weeks and this is really efficient. I've fixed between 300 and 400 bugs in GDAL...

 

So I just gave it a try with PostGIS, concentrating on liblwgeom, since it builds nicely in oss-fuzz environment

(plain "make" in top repository fails in oss-fuzz from some reason I haven't investigated)

As an example, I've created 2 fuzzers, one for lwgeom_from_wkb() and the other one for lwgeom_from_wkt().

More could be done, based on those examples. Left as an exercice to other developers.

 

Integration of a software with oss-fuzz is made of 2 parts:

- fuzzer entry points must be in the project repository : https://github.com/rouault/postgis/tree/ossfuzz/fuzzers

- a metadata file (project.yaml), a Dockerfile (download needed packages & PostGIS source code) and

a "bootstrap" build.sh script must be integrated in OSS-Fuzz own repo too :

https://github.com/rouault/oss-fuzz/tree/postgis/projects/postgis

 

For now, I've done this in my own postgis and ossz-fuzz git forks as you can see, but ultimately if the

project agrees we should merge this into their respective official repos.

 

But pending that, you can already try this stuff locally

{{{

Make sure you have Docker installed

 

git clone --branch postgis https://github.com/rouault/oss-fuzz

cd oss-fuzz

 

Build the Docker image:

python infra/helper.py build_image postgis

 

Build PostGIS and the fuzzer programs with the address sanitizer

python infra/helper.py build_fuzzers --sanitizer address postgis

 

Run one of the fuzzer (you can try with wkt_import_fuzzer too)

python infra/helper.py run_fuzzer postgis wkb_import_fuzzer

}}}

 

My local experiments show that lwgeom_from_wkt() seems to be rather robust,

but lwgeom_from_wkb() has a few bugs. For the record, I found and fixed (in my fork) this memory leak

(memory leak = leak in a case where lw_error() is not called) in

https://github.com/rouault/postgis/commit/cf179396b719223653eee56a01189339e0abcc0d

There's also a heap buffer overflow it just detected in

ptarray_from_wkb_state /src/postgis/liblwgeom/lwin_wkb.c:367

 

So if the community is interested in a closer integration in OSS Fuzz, next steps are :

1) someone with PostGIS commit rights merges https://github.com/rouault/postgis/commit/0181a28ab01764b4e6d11a5d2ffe7edce96498c6

into PostGIS SVN (as well as the bug fix https://github.com/rouault/postgis/commit/cf179396b719223653eee56a01189339e0abcc0d while you are it)

2) interested core PostGIS developers give me a @gmail.com email, so I add it in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/project.yaml

This way they will then have access to the bug reports that are embargoed for

90 days (or 30 days after OSS Fuzz has found them to be fixed)

3) I then modify https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/Dockerfile to

point to PostGIS official github mirror instead of my fork

4) I then submit a pull request to https://github.com/google/oss-fuzz/ with my

https://github.com/rouault/oss-fuzz postgis branch . They may accept or not the application, but I guess they will accept.

If they don't, you can also play with it locally as I showed above. And this is strongly recommended

to do so when adding a new fuzzer for example.

5) once the project is accepted, monitor https://bugs.chromium.org/p/oss-fuzz/issues/list?q=postgis and fix the bugs !

6) add more fuzzers. Hint: in PostGIS "fuzzers" directory, "make dummyfuzzers" to check that your fuzzer builds.

 

Note: I don't volunteer to fix all bugs that will be found. I have already enough to do with GDAL... I wouldn't

mind if someone wants to be the declared maintainer in oss-fuzz projects/postgis/project.yaml and projects/postgis/Dockerfile

 

Note 2: if you look closely at https://github.com/rouault/postgis/blob/ossfuzz/fuzzers/wkb_import_fuzzer.cpp

you will notice that it is a bit messy since it stubs GEOS and geod_ symbols. This is due to the fact

that OSS-Fuzz requires that the fuzzer programs are completely statically linked, and Ubuntu doesn't ship

with static builds of geos (actually just libgeos.a but no libgeos_c.a) as far as I can see.

All this could be improved by adding a download of GEOS source code in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/Dockerfile and building it manually in

https://github.com/rouault/oss-fuzz/blob/postgis/projects/postgis/build.sh

 

Thoughts ?

 

Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel

 


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Even Rouault-2

On vendredi 7 juillet 2017 18:46:31 CEST Regina Obe wrote:

> Okay Even sold me on this and convinced me it's not as much work as I'm

> expecting it to be. I guess I'll step forward.

>

>

>

> I'll submit a request to OSS fuzz hopefully this weekend for the continuous

> integration service and pull in Even's setup.

 

OK, if you submit the oss-fuzz part, don't forget to change the repo URL to the PostGIS official one instead of my fork I temporarily used for my local experiments. And add the emails of folks that want to have access to the bug reports in project.yaml (bug reports are embargoed to the public for 90 days)

But I can take care of submitting the oss-fuzz part if you want.

The only part I can't do myself is committing the PostGIS part.

 

>

> Even did you by chance see anything funny with Kmeans?

 

Hum, I'm not familiar with Kmeans, and as I only created fuzzer entry-points for lwgeom_from_wkt() and lwgeom_from_wkb(), oss-fuzz for now will only find issues in those parts of the code

 

> That cunit thing is

> driving me nuts cause it fails 25% of the time on windows (more on 32-bit

> runs)with some crash.

 

My generic solution for this is : Linux + valgrind, or a -fsanitize=address build (assuming this isn't a Windows specific issue, but something that's just hidden on Linux)

Isn't there a -fsanitize=address enabled config for PostGIS ? I didn't try mysefl. I guess that would probably require explict LD_PRELOAD'ing the libasan.so since Postgres will not by default by built with this flag

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Sandro Santilli-3
RE: fuzzers and Google buying us all

I received a few (~5) email notification about bugs found by the
fuzzer. But when I clicked on the links I got a permission
denied. Supposedly, I'd have to create an account on Google, and be
given permission to read that report. Is this correct ?

Can we get those fuzz tests be run by our own bots ?
Like drone ? Drone is already docker based, if that was the problem...

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Regina Obe-2
Yah I  think your email has to be a gmail. I can see them, but my account is under gmail.

And yes I think we can setup locally if we want.  I think Even had experimented with that.
We'd probably want to setup locally anyway so we can test out changes before we add to our fuzz list.  
I haven't done the leg work to figure out how to set up locally though and not sure when I'll have time to do that.
Strk -- if you want to take a stab at it, I'd be so happy :)


Thanks,
Regina



-----Original Message-----
From: postgis-devel [mailto:[hidden email]] On Behalf Of Sandro Santilli
Sent: Wednesday, July 12, 2017 4:23 PM
To: PostGIS Development Discussion <[hidden email]>
Subject: Re: [postgis-devel] PostGIS (actually liblwgeom) integration with oss-fuzz

RE: fuzzers and Google buying us all

I received a few (~5) email notification about bugs found by the fuzzer. But when I clicked on the links I got a permission denied. Supposedly, I'd have to create an account on Google, and be given permission to read that report. Is this correct ?

Can we get those fuzz tests be run by our own bots ?
Like drone ? Drone is already docker based, if that was the problem...

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel

_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Mateusz Loskot
On 13 July 2017 at 00:38, Regina Obe <[hidden email]> wrote:
> Yah I  think your email has to be a gmail.

https://github.com/google/oss-fuzz/blob/master/docs/faq.md#why-do-you-require-an-e-mail-associated-with-a-google-account

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Regina Obe-2
I should have read that.  Anyway I didn't care since my email is a gmail account.

However it does say, any email will work, but the email has to be associated with a google account.  

So Sandro, if you want to bother, I guess you just got to create an account

https://accounts.google.com/SignUp?hl=en

and say you prefer to use your own email.

I actually have that setup for some of my other email addresses that aren't gmail hosted that clients assigned me so I could share google docs with them using my client specific email address.

Thanks,
Regina




-----Original Message-----
From: postgis-devel [mailto:[hidden email]] On Behalf Of Mateusz Loskot
Sent: Wednesday, July 12, 2017 7:29 PM
To: PostGIS Development Discussion <[hidden email]>
Subject: Re: [postgis-devel] PostGIS (actually liblwgeom) integration with oss-fuzz

On 13 July 2017 at 00:38, Regina Obe <[hidden email]> wrote:
> Yah I  think your email has to be a gmail.

https://github.com/google/oss-fuzz/blob/master/docs/faq.md#why-do-you-require-an-e-mail-associated-with-a-google-account

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net _______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel

_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Mateusz Loskot
On 13 July 2017 at 05:00, Regina Obe <[hidden email]> wrote:
> I should have read that.  Anyway I didn't care since my email is a gmail account.

Yes, same here.

Regina, could you please add my [hidden email] to postgis/project.yaml ?

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Even Rouault-2
In reply to this post by Regina Obe-2

On mercredi 12 juillet 2017 18:38:24 CEST Regina Obe wrote:

> Yah I think your email has to be a gmail. I can see them, but my account is

> under gmail.

>

> And yes I think we can setup locally if we want. I think Even had

> experimented with that. We'd probably want to setup locally anyway so we

> can test out changes before we add to our fuzz list. I haven't done the leg

> work to figure out how to set up locally though and not sure when I'll have

> time to do that. Strk -- if you want to take a stab at it, I'd be so happy

> :)

 

There are 2 different things :

 

- reproduce locally a bug found by oss-fuzz. You can just build the dummy fuzzer in PostGIS by doing

 

cd fuzzers

make dummyfuzzers

and it generates /tmp/wkb_import_fuzzer and /tmp/wkt_import_fuzzer

 

Then download the reproducer test cases from the oss-fuzz ticket and do

 

/tmp/wkb_import_fuzzer the_file (or /tmp/wkt_import_fuzzer the_file depending on which fuzzer found the issue)

 

Possibly under Valgrind, or with a PostGIS build configured with

CFLAGS="-fsanitize=undefined,address", so as to catch the issues that don't systemetically translate to crashes.

 

 

- fuzz the code yourself. Then you need to use the oss-fuzz Python scripts that rely on Docker underneath. See instructions in fuzzers/README.TXT

 

Even

 

>

>

> Thanks,

> Regina

>

>

>

> -----Original Message-----

> From: postgis-devel [mailto:[hidden email]] On Behalf

> Of Sandro Santilli Sent: Wednesday, July 12, 2017 4:23 PM

> To: PostGIS Development Discussion <[hidden email]>

> Subject: Re: [postgis-devel] PostGIS (actually liblwgeom) integration with

> oss-fuzz

>

> RE: fuzzers and Google buying us all

>

> I received a few (~5) email notification about bugs found by the fuzzer. But

> when I clicked on the links I got a permission denied. Supposedly, I'd have

> to create an account on Google, and be given permission to read that

> report. Is this correct ?

>

> Can we get those fuzz tests be run by our own bots ?

> Like drone ? Drone is already docker based, if that was the problem...

>

> --strk;

> _______________________________________________

> postgis-devel mailing list

> [hidden email]

> https://lists.osgeo.org/mailman/listinfo/postgis-devel

>

> _______________________________________________

> postgis-devel mailing list

> [hidden email]

> https://lists.osgeo.org/mailman/listinfo/postgis-devel

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Regina Obe-2

Even,

 

Thanks for the input.  I managed to create a docker build I could bash into using the oss-fuzz plain build and some logic I borrowed from strks postgis-docker.

 

I was able to get as far as:

 

cd fuzzers

make dummyfuzzers

and it generates /tmp/wkb_import_fuzzer and /tmp/wkt_import_fuzzer

 

then ran /tmp/wkb_import_fuzzer test-case-file-from-bug-report-here

 

and that segfaulted as expected.  Then I made changes to ptarray_is_closed_2d and ptarray_is_closed_3d  and tested with my new version

and didn't get a segfault anymore, though I'm not confident with my solution so I'll commit that into my own repo for strk and pramsey to inspect.

 

The regress still pass with the change I made so I guess that's promising

 

That said, when trying to build with

CFLAGS="-fsanitize=undefined,address",

 

My configure just gives error

 

checking for gcc... gcc

checking whether the C compiler works... no

configure: error: in `/postgis-trunk':

configure: error: C compiler cannot create executables

See `config.log' for more details

 

So I'm missing something here, perhaps a typo.

 

Thanks,

Regina

From: Even Rouault [mailto:[hidden email]]
Sent: Thursday, July 13, 2017 5:19 AM
To: [hidden email]
Cc: Regina Obe <[hidden email]>
Subject: Re: [postgis-devel] PostGIS (actually liblwgeom) integration with oss-fuzz

 

On mercredi 12 juillet 2017 18:38:24 CEST Regina Obe wrote:

> Yah I think your email has to be a gmail. I can see them, but my account is

> under gmail.

>

> And yes I think we can setup locally if we want. I think Even had

> experimented with that. We'd probably want to setup locally anyway so we

> can test out changes before we add to our fuzz list. I haven't done the leg

> work to figure out how to set up locally though and not sure when I'll have

> time to do that. Strk -- if you want to take a stab at it, I'd be so happy

> :)

 

There are 2 different things :

 

- reproduce locally a bug found by oss-fuzz. You can just build the dummy fuzzer in PostGIS by doing

 

cd fuzzers

make dummyfuzzers

and it generates /tmp/wkb_import_fuzzer and /tmp/wkt_import_fuzzer

 

Then download the reproducer test cases from the oss-fuzz ticket and do

 

/tmp/wkb_import_fuzzer the_file (or /tmp/wkt_import_fuzzer the_file depending on which fuzzer found the issue)

 

Possibly under Valgrind, or with a PostGIS build configured with

CFLAGS="-fsanitize=undefined,address", so as to catch the issues that don't systemetically translate to crashes.

 

 

- fuzz the code yourself. Then you need to use the oss-fuzz Python scripts that rely on Docker underneath. See instructions in fuzzers/README.TXT

 

Even

 

>

>

> Thanks,

> Regina

>

>

>

> -----Original Message-----

> From: postgis-devel [[hidden email]] On Behalf

> Of Sandro Santilli Sent: Wednesday, July 12, 2017 4:23 PM

> To: PostGIS Development Discussion <[hidden email]>

> Subject: Re: [postgis-devel] PostGIS (actually liblwgeom) integration with

> oss-fuzz

>

> RE: fuzzers and Google buying us all

>

> I received a few (~5) email notification about bugs found by the fuzzer. But

> when I clicked on the links I got a permission denied. Supposedly, I'd have

> to create an account on Google, and be given permission to read that

> report. Is this correct ?

>

> Can we get those fuzz tests be run by our own bots ?

> Like drone ? Drone is already docker based, if that was the problem...

>

> --strk;

> _______________________________________________

> postgis-devel mailing list

> [hidden email]

> https://lists.osgeo.org/mailman/listinfo/postgis-devel

>

> _______________________________________________

> postgis-devel mailing list

> [hidden email]

> https://lists.osgeo.org/mailman/listinfo/postgis-devel

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Even Rouault-2

On jeudi 13 juillet 2017 18:20:09 CEST Regina Obe wrote:

> Even,

>

>

>

> Thanks for the input. I managed to create a docker build I could bash into

> using the oss-fuzz plain build and some logic I borrowed from strks

> postgis-docker.

>

>

>

> I was able to get as far as:

>

>

>

> cd fuzzers

>

> make dummyfuzzers

>

> and it generates /tmp/wkb_import_fuzzer and /tmp/wkt_import_fuzzer

>

 

To be clear, you can also do that outside of the oss-fuzz docker environement and use your regular dev environment.

 

I rarely use the oss-fuzz Docker environement (just to check that the fuzzer programs work well when I create them) and never bash into it.

 

> That said, when trying to build with

>

> CFLAGS="-fsanitize=undefined,address",

>

>

>

> My configure just gives error

>

>

>

> checking for gcc... gcc

>

> checking whether the C compiler works... no

>

> configure: error: in `/postgis-trunk':

>

> configure: error: C compiler cannot create executables

>

> See `config.log' for more details

>

 

Works for me. Perhaps your gcc version is too old and doesn't support the sanitizers.

 

And to build the fuzzers with a liblwgeom built with the sanitizers, you also need to do:

CXXFLAGS="-fsanitize=undefined,address" make dummyfuzzers

 

Note: I noticed that the *.sh scripts in fuzzers/ don't have execute permissions.

You need to

svn propset svn:executable on fuzzers/*.sh

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Sandro Santilli-3
In reply to this post by Regina Obe-2
On Wed, Jul 12, 2017 at 11:00:33PM -0400, Regina Obe wrote:

> So Sandro, if you want to bother, I guess you just got to create an account
> https://accounts.google.com/SignUp?hl=en
> and say you prefer to use your own email.

I don't want to give Google more than it has already.

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Mateusz Loskot
On 14 July 2017 at 21:05, Sandro Santilli <[hidden email]> wrote:
> On Wed, Jul 12, 2017 at 11:00:33PM -0400, Regina Obe wrote:
>
>> So Sandro, if you want to bother, I guess you just got to create an account
>> https://accounts.google.com/SignUp?hl=en
>> and say you prefer to use your own email.
>
> I don't want to give Google more than it has already.

Isn't sparing an e-mail address worth money PostGIS (Your!) customers
would thankfully save thanks to your product being offered
lighter of dozens (or hundreds) of bugs?
The answer is no, I guess.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Mateusz Loskot
On 14 July 2017 at 21:29, Mateusz Loskot <[hidden email]> wrote:

> On 14 July 2017 at 21:05, Sandro Santilli <[hidden email]> wrote:
>> On Wed, Jul 12, 2017 at 11:00:33PM -0400, Regina Obe wrote:
>>
>>> So Sandro, if you want to bother, I guess you just got to create an account
>>> https://accounts.google.com/SignUp?hl=en
>>> and say you prefer to use your own email.
>>
>> I don't want to give Google more than it has already.
>
> Isn't sparing an e-mail address worth money PostGIS (Your!) customers
> would thankfully save thanks to your product being offered
---------------------------------money, I meant money, for sake :)
> lighter of dozens (or hundreds) of bugs?
> The answer is no, I guess.



--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PostGIS (actually liblwgeom) integration with oss-fuzz

Sandro Santilli-3
In reply to this post by Mateusz Loskot
On Fri, Jul 14, 2017 at 09:29:17PM +0200, Mateusz Loskot wrote:

> On 14 July 2017 at 21:05, Sandro Santilli <[hidden email]> wrote:
> > On Wed, Jul 12, 2017 at 11:00:33PM -0400, Regina Obe wrote:
> >
> >> So Sandro, if you want to bother, I guess you just got to create an account
> >> https://accounts.google.com/SignUp?hl=en
> >> and say you prefer to use your own email.
> >
> > I don't want to give Google more than it has already.
>
> Isn't sparing an e-mail address worth money PostGIS (Your!) customers
> would thankfully save thanks to your product being offered
> lighter of dozens (or hundreds) of bugs?

Customers are welcome to register an account and receive those emails.
The official way to report bugs is via the bug tracker, those are
always welcome.

Speaking of which, today I filed a pull request for a project
(ansible) and a nice bot automatically pointed out things to be
taken care of. It was very useful. Can a bot file trac tickets
for problems found by oss-fuzzer ?

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/postgis-devel
Loading...