[Grok-dev] why are REST and JSON tests so different?

[Martijn Faassen]

Hey Brandon,

Brandon Craig Rhodes wrote:
[snip]
> * The JSON tests never seem to actually test a real, live method call
>   over a browser request to show that things actually work.  All of the
>   tests in `tests/json` seem to either test bare components (as in
>   `json_layers.py`) or test various sorts of errors that programmers can
>   make in setting up their `grok.JSON` subclass.  I don't actually see a
>   test of whether a JSON HTTP request actually works. :-)

There is one I can find, but it's in ftests/security and just tests 
security aspects and JSON is only tested incidentally. It'd be nice if 
we could get some more tests in ftests for json in particular.

> * Nearly all JSON tests are unit tests, while nearly all REST tests are
>   functional tests.  I am not sure what the logic was behind testing
>   them each in this way; it seems, again, like both would have wanted
>   unit tests for anything that checks whether the classes works, and
>   then several ftests that actually throw HTTP requests at a little Grok
>   web application and see if the answers come back correctly.

I couldn't think of much to do with non-functional tests with REST when 
I wrote them, and I was lazy. The whole point of REST is to see whether 
something actually gets published properly on GET, POST, PUT and DELETE, 
and only an integration test can really do that. There's very little 
that REST views do otherwise.

For JSON the situation is at least slightly different as the JSON 
encoding aspect can be tested. In fact once we get to tackling 
https://bugs.launchpad.net/grok/+bug/314836
we could do that in unit tests too. That said, I don't think doing it 
with ftests there would have been a big problem either.

I think this situation just grew historically; I did what seemed right 
testing REST and we did what seemed right when testing JSON before then. 
When JSON support was added we had less experience with what seemed 
right. :)

> * When REST wants to test layers it uses URLs with ++'s all over them,
>   as I would expect.  When JSON tests layers, it does lots of object
>   adaptation.  Mightn't we want to standardize on one way or the other
>   for testing layers/skins with both of them, so that a similar feature
>   in two places in Grok has similar-looking tests in both places?

My conclusion based on your question is that we need more JSON ftests, 
as they're easier to read in this case and they actually test 
integration aspects we need. There is a slight performance benefit of 
doing non-functional tests, but I don't think we're winning much there. 
non-functional tests could be written for the pure encoding aspect of 
JSON, but otherwise I think it'd be nicer if we got functional tests.

Regards,

Martijn