Visualizing Movie Data | VMD_07 | Movies by words

All films that we have seen in 2015 (and earlier) make use of a film script. What are the most often used words in this scenario? To find out, I need to look for movie scripts. I found them in the ‘IMSDb’ (Internet Movie Script Database). The first script that I found was the script of ‘Tamara Drewe’. Certainly not the best movie we’ve seen but that doesn’t matter much in this case. I loaded the script into TextEdit. Then I imported the complete script of ‘Tamara Drew’ into Processing. In Processing I used Ben Fry’s Treemap Library. In this version I have everything inverted. And I have added a sixth layout algorithm: PivotBySize. The words ‘the, a, is, you, to, tamara, nicholas, beth, and, in, i’ appear to be most often into this script.
VMD_07_01

Here I used the script of ‘The Master’. The starting point is now to use a gradient in the background. I do that by placing horizontal lines. This is a less advanced version. In this script are ‘the, freddie, to, you, and, a, master’ the most frequent words.
VMD_07_02

In this version, I’ve refined the gradient. I also changed the font to Avenir Light. And I turned off the frames. In this script of ‘Interstellar’ are ‘the, cooper, a, to, or, and, it, is, in’ the most common words.
VMD_07_03

This variation is only a variation in the code. I have programmed a few things slightly more functional. Used the script of the film ‘Blow’. The words ‘the, and, a, you, george, to, is, i, of, are most frequently used in the script.
VMD_07_04

I started again with studying the Treemap library. I have added two new classes. The BinaryTreeLayout and PivotByMiddle class. And I used a less harsh color scheme. I used the script from the film ‘Foxcatcher’. The most frequent used words are ‘the, to, mark, du pont, a, of, dave’.
VMD_07_05

A very colorful variation. The largest number that can be formed by w (width) in this version is 190 (pixels). All numbers that are higher do not have any influence on w for w can never be higher. Suppose I want to use six colors than that would be 190 divided by 6 colors (rounded up) is 32. So I can choose 6 shades of color in HSB color mode every 32 colors. As a script I used ‘Fargo’ here. The most frequent words are ‘the, a, and, to, margin, or, you, his’.
VMD_07_06

There are no scripts available anymore from movies that we have seen in 2015. So I have to search for an alternative. Films that we have seen but not in 2015. Also the range of colors has to be more advanced. If I want to use the entire 360 HSB range than I have 203 pixels for w (width) available. I have to divide 203 by 12 in this release. That’s 16. In the end this calculation doesn’t work. So I made increments of 40 in the HSB color mode. The script is from ‘12 Years a Slave’. Most common words are ‘the, a, to, and, solomon, or, is, in’
VMD_07_07

In this final version I have used a better color scheme. I also used the script of ‘Inglourius Basterds’. Most common words are ‘the, a, to, and, in, or, you, i, his’.
VMD_07_08

Advertisements

Visualizing Movie Data | VMD_02 | Time Series

A dozen years ago I heard the word ‘ubiquitous’ for the first time. I wondered for what it stands for. Looked it up and the word ‘ubiquitous’ means: present, appearing, or found everywhere. So these time series graphs are a type of graphs that you can find anywhere. Because this project is about visualizing our movie data I need three or more data sets. The idea is that I will compare these data sets with our own data set. I hope to find out how our qualifications relate to, for instance, the IMDb (Internet Movie Database), Metacritic and/or Rotten Tomatoes. Suppose I would like to see the first one hundred films compared to results of these websites I should be able to draw the necessary conclusions. This is going to be a lot of handwork. But that’s okay because I’am in a learning process.

I could imagine that you have the numbers 1 to 10 on the left side of the graphic. And at the bottom are all the film titles. That seems logical. But it is not. Movie titles may be very long. For example: ‘A Pigeon Sat on a Branch Reflecting on Existence’. So you would expect the film titles to the left side of the graph. And then, the numbers 1 to 10 at the bottom. At this moment I think the best solution would be if you place your mouse cursor on a data point that the movie title is displayed at that point. But maybe I run too much ahead of myself. I have now read the original data of Ben Fry’s Time Series chapter in the program and I changed the display format.
VMD_02_01

Let me concentrate on the data. The first thing you notice about the IMDb, Metacritic and Rotten Tomatoes reviews is that they work with floats. Our own movie data works also with floats but the end results in ints. So I actually have to run all 100 film programs again and see what the endresult is using float’s. When I have those results, I have to type them in a text file. And then I do the same with the results of IMDb, Metacritics and Rotten Tomatoes. I left out Metacritics in the end. It sometimes happens that we have seen a film but that it is not found on IMDb or Rotten Tomatoes. In that case, the film gets a zero. The first thing I noticed in our chart, which uses our own data, is that it looks quite messy. There is not really some logic to find in the positioning of the points. The reason for this is that our films are chosen randomly. This results in random positions for the positioning of the set of points. The sequence is the real sequence of the first 100 films we have seen in 2015 though. Furthermore, the points are positioned at the bottom. This is caused by the largest value in the other data series. Our data set ranges from 0.0 to 10.0. While the other two data sets a range from 5.1 to 46.4. Therefore these other two sets have still to be adjusted. But I do not have the right data for them yet.
VDM_02_02

At this moment I have added all the scores from all the IMDb and Rotten Tomatoes. I can now on hit the “]” key and the “[” key to go through the three different graphs. It all looks a bit scarce. But you do get an impression of how the scores are distributed. I’ve also added titles as a placeholder (We, IMDb and RT (Rotten Tomatoes)).
VDM_02_03

I have increased the number of films to 150. It now looks somewhat less scarce. Eleven films from Rotten Tomatoes are not evaluated. That makes them stand to zero. At the bottom chart of the chart. However, these films are evaluated on IMDb and by us.
VDM_02_04

At the bottom, I added the amount of films we have seen in numbers. I also reduced the white background space slightly. This ensures that everything is shown less cramped in the display window. It would even be better when you could read the titles of the movies instead of our numbering. But perhaps I can add that at a later stage. And perhaps not at all. Maybe. Because after all these graphs are only about comparing our voting behavior with IMDB and RT. A quick conclusion about it teaches that our differences are slightly wider spread. It ranges from 3.3 to 5.9 points. IMDB ranges from 4.1 to 9.3. Rotten Tomatoes series go from 4,5 to 9,8 (if you do not count the 0.0).
VDM_02_05

I have added horizontal and vertical grid lines that may be helpful to compare the data points better. On the left side of the graph are now the scores of 0.0 to 10.0 displayed. And as a result, there is no need for the positioning of additional tickmarks. The horizontal and vertical lines do their work instead. I think that score numbers are displayed too long. I have now four digits after the point because we are working with floats. The function ceil does not help in this case. Because that rounds everything off upwards. Floor rounds everything downwards. The feature I’ve used now is nf. This means that there is just one number after the point shown. I use two versions of the Futura. Futura Medium and Bold. Furthermore, I also labeled the numbers. That makes the chart clearer.
VDM_02_06

I now go ahead replacing the points with a line. Actually this is a bit rubbish. The scores of the films have nothing to do with each other. Each score of a film state is a value on its own. So there is no mutual connection with a line necessary. But as a variation it is perhaps interesting. I also changed the colors. The white field is replaced with a dark gray. Because then the colored lines stand out better.
VDM_02_07

In this version all scores are displayed on top of each other to see where the differences are. The title of the data sets should change with it if you choose another data set. But I don’t like it anyway. It is a poor and chaotic whole. So this seems to be not a good option.
VDM_02_08

I now have retrieved some items from one of the earlier sessions. The line connections remained blue and the points themselves are white. The points are most important so they are allowed to stand out. I’ve made them a little smaller. This has as a result that (when points are close to each other) they overlap each other less.
VDM_02_09

This proposal introduces rollovers. I now get feedback that I already can see on the x and y axes but much more precise. But actually you would like to see the movie title when your cursor is at a data point. I think I’m going to do that at a later stage. But I am unsure about it. I think it’s it’s more important that I get some sense of what you can do with the data.
VDM_02_10

I do have the feeling that the lines have become too dominant. Especially now that you’re getting direct feedback on the cursor. The lines are no longer functional. I will also try if I can make the middle block more squared. You lose that  the smaller rectangles are not square anymore. However, it does create more room in the width. I also reduced the proximity of the cursor and increased the point size of 10 to 12. And Futura Bold is used for the values under the cursor.
VDM_02_11

Replacing vertex in drawDataLine by curveVertex actually does not make much sense. The data points are most of the time so close together that no fluid line between the points can be made. But if you make a plane field to the lower right point right and the lower left point it makes more sense and it gives a different picture. The question then is whether the horizontal and vertical lines are still functional. So I have them  removed. I think this looks better than all the previous versions. And along with the feedback you get when you stand with your cursor on a data point it looks just fine.
VDM_02_12

I have made the background of the chart the same color as the background color. That gives a completely different picture. I initially had accentuated the vertical lines. But I think the horizontal lines can better be accentuated. These lead you too much more meaningful data. I have given the horizontal lines 50% transparency in the beginning. But afterwards I got a better result by decreasing the line width to 0.5 pixels. Which is basically logically impossible.
VDM_02_13

It seems silly to transform this graph to a bar graph. I must then let the program draw rectangles instead of one flat plane. But then I have a problem. Because I have 150 bars in a width of 600 pixels. This means that the width of one bar can be a maximum of 3 pixels or less. At 4 pixels, the total lower surface is filled again by overlapping bars. But with 3 pixels I think it’s just about acceptable and it even has some form of sophistication.
VDM_02_14

As a last proposal I introduced tabs for the three different data sets. But I found the Futura Bold far too heavy in these white tabs. So I opted for the Futura Medium.
VDM_02_15

Now I have to do a few more things. The white area behind the title is way too loud and is almost visually independent of the graph. Plus the bar chart layout is not the best I’ve seen so far. As a final detail I go back to the design of VDM_02_12. I now only use the Futura Medium. I also adjusted the color. I chose red and green. Two distinctly different colors. The strong contrast between the two colors allows the separation-line between the two planes extra stand out. And thus it seems to me that this session is finished. But there is one more thing.
VDM_02_16

I have made a very simple animation of the three datasets. The datasets of us, the IMDb and Rotten Tomatoes interpolate their points. Unfortunately, the interactive version is not available. I captured the animation so that there is atleast something to see.
VDM_02_17

 

Visualizing Movie Data | VMD_01 | Mapping Movie Data

In 2008 we began reviewing movies. In the beginning we were doing that using points that lay between 0 and 10. At a later date we used a more narrative way. That led to detailed reviews of movies that we posted on Facebook. But over time was that too much work and it cost too much time so I decided to introduce a more accurate way of reviewing the movies. I started on 5 January 2015 to use 13 categories for each movie. Storyline, originality, cinematography, involvement, sound, editing, educational, title design, acting, interesting, unusual, exciting and superior. Every category earns a score between 0 and 10. A simple Processing program add’s all points together. And the result of this addition is divided by the number of categories. Then you get an average point for one movie. The aim of Visualizing Movie Data is to give us more insight into the choices we make in evaluating a film. Ultimately, this should tell more about ourselves than about the films. If that’s true, that would be a positive spin-off. Through Visualizing Movie Data I try different ways of visualizing data collected by us. And I get help from Ben Fry’s book ‘Visualizing Data’ published by O’Reilly Media Inc. Contrary to the Generative Design Variations project Visualizing Movie Data is not about making as many as possible variations. Now it is the intention that the movie data is used as functional and as basic as possible. I will skip any form of decoration. In this chapter I try to use a very simple and basic way of reading, displaying and interacting with a number of small data sets. These datasets consist partly of the data we found in our movie reviews.

I started looking for a world map. Eventually I found a world map that uses the Mercator projection. Mercator projection is a conformal cylinder projection with large surface deformations at higher latitudes. In this projection Europe is slightly larger which is an advantage because a lot is going to happen in this ‘small’ part of the world. Perhaps it is ultimately necessary to use a separate map of Europe as an insert. But I don’t know that at this moment. I decided to omit all color because I want to use the color for the markers of the countries. At this stage I only display a world map.
VMD_01_01

This is a first version in which the program reads coördinates from a text file to place red dots. I wanted to know if that worked. And If that works then it should also work with exact coördinates.
VMD_01_02

Now I need to gather a list of all 46 countries that have produced films. I happened to find a two-letter code list of countries through the International Organization for Standardization (ISO 3166 Country Codes). These abbreviations I can use to display in a later stage. The overall list is much too extensive so I have to select only those countries that have made films that we actually have seen from the beginning of 2015. If you look at this visualization there are a few things that stand out. In Europe it is very busy. Some countries are completely covered by a red dot. And some dots overlap other spots. Actually, it’s a mess. On all other continents there are only a few spots. It looks empty. Further, the overall color of the image looks too dark. I also have made an outline version of the dots. But this does not solve the problem. In Europe the small countries are slightly more visible but it is a minimal improvement and no real solution.
VMD_01_03

This version I’m going to make an insert for Europe. At least so I thought at that time. After I had made a rough sketch it actually delivered more problems than a solution. In the first place Europe is out of proportion when you compare it with the rest of the world map. Which is the case anyway at each world map. An exception is the projection of Goode. That has is an equal-area map projection. And in the second place an insert covers always a portion of the world map. Making that to be moved again. And actually you create five continents, which are all out of proportion. The question is whether that’s good. So it seemed best to temporarily leave everything as it is. And I solve problems when they are relevant. The dots are slightly reduced and everything becomes pretty clear. I have also added a title plus additional information which makes it even more complete.
VMD_01_04

A few years ago I have worked on patterns and photography in Processing. I could apply one of those patterns on the world map. Since a realistic world map is not possible anyway, it is just as good to make an abstraction of the world map. The question is now: is abstraction decoration? The world map is now completed with dots with a diameter of 4 pixels. The locations are displayed (as much as possible) in the middle of the country. Although the middle in some countries is hard to find. Where is the middle of the USA with its territories and various possessions?
VMD_01_05

Those dots are of course meaningless. They only give the central locations of countries where movies are made. It would for instance be more helpful when you could see where the most films are made. This would be able when you could vary the size of the dot. Large dot is a lot of movies. Small dots are a few films. This is a version that uses random generated numbers.
VMD_01_06

This version uses our movie data. The size of the dot determines the number of films from that country. It is immediately clear that most films come from France, USA, UK and Germany. This of course says nothing about the quality of the movie. It only shows information about the quantity. And it is only partly the truth as we will see later. And how will this visualization look like at the end of 2015 or 2016 as we have seen more movies. It was a surprise that France plays such a leading role in the field of film production. But it also raises immediately questions. Maybe we’re being manipulated? Might there be another variable which makes that France appears so high in the movie production?
VMD_01_07

It is also possible to interpolate between two colors, and make all dots the same size. I go for the low numbers in red and green for the high numbers. But I find that this version does not really show the smaller differences very well. In fact, there are only two green dots and two interpolations between green and red tending towards brown. The rest is nuances in red. And what does green-ish or reddish-brown mean then?
VMD_01_08

All countries of which we have seen five or more films are green. The largest dot on the world map indicates that we have seen much more than five movies. A small dot indicates that we have seen five movies. All countries of which we have seen less than five movies are red. A larger red dot indicates that we have seen almost 5 movies. A small dot indicates that we may just have seen only one movie. In short, very complicated and not very clear. And you really need some textual information here.
VMD_01_09

Of course you do want to be more accurate in displaying this data. So I’ve changed the program in a way that the abbreviated name of the country is displayed when you get to the dot of the country with the cursor. This goes not flawless though. Sometimes the name of the country disappears under a dot. At a certain point when the countries are small and close together both country names are being activated. I made a version of the Futura Medium 12 and with Futura Bold 12. I think that the Futura Bold version is better suited because it is more readable.
VMD_01_10

I have optimized the mouse interaction. Now, there are never two countries selected at the same time and it is also true that the name of a country is always drawn last. So it can never be overwritten by a dot.
VMD_01_11

I have replaced the floating numbers with integers. And I replaced all the colors by green. I think this is a reasonable version. You can easily see from which country most films come from. Ad the for the exact quantities you get feedback from the cursor.
VMD_01_12

I’m going a little deeper into the movie productions made in France, United Kingdom, Germany and the USA. I begin with France. That data does not look very spectacular. Apparently, most of the movies are filmed in Paris. In addition there are a few film locations in the south and center of France. Furthermore it seems that of the 47 French films that we have seen only 20 are filmed on a location in France. The other 27 are made through partnerships with other countries. And the film locations are all situated outside of France.
VMD_01_13

There is a problem with the data. When I run the program I get an ArrayIndexOutOfBoundsException: 8 error. I tried to find out what causes this error. It appears when you read data from text files that are of different lengths you get an error. That seems logical because in that way the arrays can not all be filled just the same way. If you have an array with 10 lines and another with 8 lines you get this error. And that was the case when I went to adjust the French data files for the United Kingdom data files. The 8 in the error message is the number of lines that were set aside for the array.
VMD_01_14

It looks empty in Germany. But I think it all will be fine in the long-term. Also, I think I’ve done something wrong. Some movies, of course, have several film locations. If I can trace those it is guaranteed that the image is getting more interesting.
VMD_01_15

Also this US version is not really interesting. But something else is happening. If the first two characters in my text file are not unique (and thus are duplicates in the list) the positioning of the dots and text goes wrong. And because the first two characters have no further function (but apparently have influence) it might be better to keep this form of abbreviations: AA, AB, AC, AD, AE, etc., and after AZ continuing with BA, BB, BC, BD.
VMD_01_16

I have now found all film locations for all American films we have seen since January 2015. I did a much too  superficially search so I found only 18 locations. Now I have 218 locations available. A number of them will not be used because the film locations are outside the USA. And there are several movies that play at the same location. So the final list will be shorter than 218 but longer than 18. Now I need to make a list and avoid duplication. All the movies filmed in Los Angeles are to be summed up to a total. All films in New York. All films of Detroit. And this goes for all the other cities and towns in the USA. I ended up with a final list of 87 cities and villages. Los Angeles has reached the top 61 productions. Which is actually not very surprising.
VMD_01_17

I now go one level deeper. I started on a global scale. Then countries scale. And I’m now going to work on an urban scale. After some research it seemed suitable to me to use the film locations of the series ‘Breaking Bad’. I display the abstract version of the map of Albuquerque in the background. And I used a reduced version of the Breaking Bad wordmark. But that doesn’t work at all. I also find the amount of film locations insufficient.
VMD_01_18

After some more research I could trace a lot more ‘Breaking Bad’ film locations. I changed the title and background colors. But I think it’s really ugly. When something is designed simple and basically it doesn’t have to look ugly. So I have to work on that.
VMD_01_19

This is the state in which I want to finish this exercise for the time being. All locations can now be read and the amount of scenes are shown after the location name. There is a title and a subtile. You can immediately see that Walter White’s House is the film location which is most used (with 80 scenes). Then Jesse Pinkman’s House follows (with 42 scenes). Hank and Marie’s House (with 29 scenes). The DEA offices (with 27 scenes). The Car Wash (with 22 scenes), Jesse Pinkman & Jane’s House (with 21), Gus’s Laundry Service (with 20) and Los Pollos Hermanos (with 17). Too bad I can not show this in a JavaScript version because that is unfortunately not working.
VMD_01_20

A rough conclusion: Data visualization is much harder than dreaming up nice effects like I did in the previous Generative Design Variations project. In data visualization you should limit yourself in order not to come up with lots nonfunctional decoration. A lot of time must be invested in research in the beginning, during and sometimes afterwards the design phase. And it is an iterative process. There are always improvements possible after the improvement. It also takes more time than a general design job. You must be very precise and constantly looking for better data and better interpretation and visualization of the data. Data visualization has a high level of detective work.

Generative Design Variations M.5.5 The sunburst tool

First a short description from the Generative Design book: ‘The sunburst tool is a convenient way to access many important parameters when generating sunburst diagrams and demonstrates all the code in this chapter. In addition, information on folders and files is displayed when the mouse moves over the sunburst diagram. For this purpose it must be determined on which object the mouse is located (i.e., on which ring), and which arc is displayed at this angle. This is complicated by the fact that the rings are drawn with different widths using the calcEqualAreaRadius function.’ And here is the program.
M_5_5_01_TOOL

The sunburst tool program does not work in JavaScript because the controlp5 library is not supported by JavaScript. So I made a flickr album of all images I made during this session.

I intend to start again with our movie review data directory. Therefore, it is necessary to explain something. We have started watching films and reviewing them using this simple tool in January 2015. Our film review data tool consists of thirteen components that all represent film quality on a scale of 0 to 10. The components are: Storyline, Originality, Cinematography, Involvement, Sound, Editing, Educational, Title design, Acting, Interesting, Unusual, Exciting and Superior. When we have seen a film we give points through these components. Here is an example:
150502_131248_0002

Now I want to compare all the reviews together. Of all the films we have seen I want to see all ratings of all components next to each other in a sunburst diagram. That means I need to modify the directory structure. All films are now in one directory and they are arranged on date. But now I want all scores of Storylines (from 0-10) in one directory. Roughly, I will have to place manually 100 films in a new directory called: ‘Storylines’. And the same I have to do with the other components. Originality gets a directory with all the scores of all originality components of 100 films. Cinematography gets a separate directory with all the scores of all cinematography components of 100 films. That means that I have to put 1300 files manually in separate directories. That took me two days. Indeed, it is very boring paperwork. And if something is boring the work slows down and it takes more time.

Maybe I could have create those directories using AppleScript but I did not know how long it was going to cost me to make a script for it. Doing it all manually I was sure it was finished after two days (although I did not know that it would take me so long). Then it’s a matter of running the sunburst tool for 13 times. Feeding it with the different directories and make screen dumps. All inner rings represent 10 scores. The rings to the outside represent the 9, 8, 7, 6, 5, 4, 3, 2 and 1 scores. The outer ring represents the 0 scores. I was surprised because the result is visually as dull as the work of the past two days.
M_5_5_01_GDV_01

Because the visuals were all rather disappointing, I just looked what happened when I feeded the sunburst tool with entire file structure of Generative Design Variations. That looked better than the results of the past few days. The same, I tried with my internal accounting (bookkeeping) directory. And with the directory of my loftmatic website. It all looked very interesting. I copied the file structure of the film review data from my desktop to the Generative Design Variations folder and then I’m went to watch the last episode of Sherlock.
M_5_5_01_GDV_10

The next day I opened the Generative Design Variations file structure in the sunburst tool. I worked on that since October 2013 and it contains at this moment 36260 items. It looked very different as the results from the day before. In fact that is right because I had added the film review data directory and files the day before. It looked even better. I worked a few hours with the sunburst tool to create variations using the Generative Design Variations directory, my accounting directory and the loftmatic website directory. That led to better results. And my work of the last two days had still been useful.
M_5_4_01_GDV_12

Looking back at this session, I must confess that the sunburst tool leads to good results if you have a complex file structure. The advantage of the sunburst tool is that anyone can use it. But that is an immediate disadvantage because you get almost always similar results. Less complex directories deliver different results than complex directories with lots of levels, folders and files but the design language is recognizable. And as a designer you don’t always want similar results. In fact I would like never to have similar results if I don’t need them. So I think that every tool that you use should have more variables to change the visualization. And that is the nice thing of a programmed tool because you can always change the code to your specific needs.

Generative Design Variations M.5.4 Sunburst trees

The second part of the sunburst diagrams takes me into the sunburst trees. The description of the Generative Design book is as follows: ‘Thanks to the preparatory work in the two classes FileSystemItem and SunburstItem, it is now easy to create a networked and tree-like representation based on the sunburst diagram. Only the SunburstItem class needs to be extended by the ability to draw a connecting line to the parent element. Bezier anchor points are also required to connect the two instances of SunburstItem with curved lines.’ That is what the Generative Design book learned me. You can find the original code here.
M_5_4_01

Because the program will not work in JavaScript I made a flickr album with all images I created during this assignment. You can find that here.
GDV_M_5_4

I copied all comments of the book version into the program itself. I always do that because that is one way to get into it. Changed the background color to black. I almost changed nothing in the program. Not that I could change much because that FileSystemItem class of Ben Fry looks pretty daunting (or let me call it challenging). This is how my sunburst tree looks now.
M_5_4_01_GDV_01

Oh… I forgot to give the dots a color. In fact they have a color but you wont see them because the dots are the same color as the background. Fixed that. The next thing I am going to do is to update our film directory with three new films. And I will also change the order of the directory. In the earlier version the top-level of the directory was filled with films who received 10 points. I will reverse that. So the top-level (the inner part of the circle) will be filled with films who have scored 0 points and the lower level (the outer part of the circle) will be filled with films which scored 10 points. It makes a minor difference in the shape of the diagram if you compare it with the M.5.3 tree diagrams chapter. Line quality is terrible. Have to find a way to improve that.
M_5_4_01_GDV_02

Spend a lot of time to check how the colors are set. It is still not 100% clear to me how that exactly works. I think I could make a special assignment about that. But I will not. I think also ben Fry’s FileSystemItem class needs some more attention from me. Finally I did manage to improve the line quality.
M_5_4_01_GDV_03

My question is now… why are the outside dots always filled with the same color? Another strange thing are the fileArcScale and folderArcScale variables. Are they in use? I have comment them out and the program behaves normal. I don’t see any change in the diagrams. And the program doesn’t make use of arcs. I kept the brightness of lines and folders on 100. When I put dotBrightnes to zero in the initialization it still displays the same. It doesn’t make any difference. But when I change dotSize during the initializing phase it does make a difference. Setting hueStart to zero and hueEnd to 360 makes everything red except for the lines and the directories and subdirectories. Setting hueStart and hueEnd both to 360 makes  also everything red. Setting hueStart to 359 and hueEnd to 360 gives me the total range of hue’s.
M_5_4_01_GDV_04

I have replaced calcEqualAreaRadius (which should return radii to have equal areas in each depth) for calcAreaRadius (which returns radii in a linear way). It is not clear to me why calcEqualAreaRadius does not returns equal area radii while calcAreaRadius does. This gives more room to see the individual dots. Also increased the size of the diagram. The bezier version of this diagram will be the first version of the diagram which may look nice but is totally not functional.
M_5_4_01_GDV_05

Another totally nonfunctional diagram variation. I only show the bezier version because the line version stayed the same as in the other earlier version. I did not work on that.
M_5_4_01_GDV_06

And because I did not work on the line version I have spent some more time on it. I finally ended up with a variation that shifts all sunburst items in the line version and in the bezier version. Which is again a totally not functional diagram.
M_5_4_01_GDV_07

I adjusted the line thickness and the begin and end of the lines. Modified the transparency a bit. Small adjustments on the size of the diagram to let it better fit into the display window.
M_5_4_01_GDV_08

Pulled everything outside the border of the diagram. Again not very functional because you cannot trace the links between folders files and directories.
M_5_4_01_GDV_09

Default settings of lines and some extravagant adjustment of dots and transparency.
M_5_4_01_GDV_10

Generative Design Variations M.5.2 Reading data from the hard drive

This part of the tree diagram chapter is a bit different from the other ones. It’s about reading data from your own hard drive. I quote the Generative Design book: ‘The files in a folder on your hard drive serve as the data basis for the tree structures in this chapter. In the section ‘Collage from collection,’ we demonstrate how several files from one folder could be loaded. This exercise differs from that example in two ways. First, the ‘tree-structure’ within a folder is required for visualisation, meaning that for each folder, all subordinated files and folders have to be traversed recursively. Second, we are no longer interested in the actual file contents but rather only in information about the files, such as their size. In order to load and display a folder structure, a new data class FileSystemItem is created. Each instance of this class can contain any number of FileSystemItem instances. An output method in the FileSystemItem class is needed for an initial simple presentation of the processed tree structure. The printDepthFirst () function recursively processes and displays the entire tree structure in the Processing console with line indentation, which indicates the depth. The tree is processed according to the principle of depth-first search: the algorithm first goes into the depth of the leaves and then branch by branch into the width. The principle of depth-first search is the ‘natural’ way that a recursively defined function processes a hierarchical structure.’ That is what the Generative Design book tells me. If you would like to see the original code you can find it here.

Now the point is… there are no graphics involved. I only get feedback in the console window of the Processing Development Environment. So I think its best to write a bit more about what I did instead of showing a tree-structure in ASCII text. I started with looking for a folder which might be interesting to read. I copied my review of films folder to the desktop and specified the path of the default folder to that film review folder. Than I let the program run. The result is a terrible mess of folders and files. It is an ordered list but it looks very chaotic. But that is because my directory of films is chaotic? Maybe it’s still a good idea to show some screen dumps. It saves me a lot of writing and it shows the idea more clearly. Here is one:

M_5_2_01_GDV_01

Looking under the FileSystemItem tab I found the class FileSystemItem. It’s a class based on code and ideas of Ben Fry. I need to go a bit deeper into that before I can say something sensible about it. FileSystemItem looks for a file or folder. Found references to sub-objects are stored in the array children. It also sorts file names alphabetically. It skips . and .. directory entries and symbolic links. It also checks if the child elements exists. Aliases or shortcuts are skipped. And than a new instance of FileSystemItem is generated for all folders and files. But I needed a much more simpler directory. So I made a fake directory. The root folder is called FakeDirectory. And it contains four FakeFolders to begin with:

M_5_2_01_GDV_02

As I noticed the output gives me two representations of the same FakeDirectory. One is called printDepthFirst and the other one is printBreadthFirst. I was not familiar with these two sorting processes so I looked it up. Depth-first search starts at the root of the directory and then it searches as far as possible along each branch before going back. Breadth-first also starts it’s search at the root directory and then it tries to find the neighbor nodes first before ging into the next level of neighbours. You can see the differences in this screendump:

M_5_2_01_GDV_03

And that is (in short) what there is to tell about this exercise. There are some other issues which I did not mention. But I think I will use that fake directory in the next assignment. Don’t know if that will work but we will see. At this moment I think it’s best to start as simple as possible. Adding complexity later when it is needed.

Generative Design Variations M.1.4 Noisy landscapes

The introduction in the Generative Design book of this assignment reads as follows: ‘The preceding example of cloud textures is easily expanded to include pseudo realistic landscapes with mountains and valley’s. Rather than coloring pixels in levels of gray, the height position of the points on a grid is modulated using the same principles. The resulting grid is only sketched here and is examined in more detail in the chapter ‘Formulated bodies’. And here is the original program:
M_1_4_01

I made a Flickr summary page. Here you can find most of the images I made during this assignment. If you click at the thumbnails of this loftmatic page you can find all programs. It was of no use to do a JavaScript conversion because the programs don’t work in JavaScript. I did not spend any time on finding out what the issue was because I would like to study Processing. Not JavaScript.

Two new things are introduced in this chapter: Processing 3D and the TileSaver class of Marius Watz. The TileSaver class is used for rendering high-resolution images by splitting them into tiles using the viewport. The program is full of public void’s which I never have used. To dive into that is a bit too much work if you ask me. So I will only concentrate on the main program. And I think that is the essence of classes. That you don’t worry about them. Just use them. The Processing 3D part is new for me. I never did anything with it until now. But in the past 15 years I have worked a lot with NewTek’s LightWave 3D. Here is an example of an animation film which Jeanne de Bont and I made with Lightwave. So the first thing I would like to do is to get all color out of this object. After that has been done I need to check what I can do with the lighting settings. But first I have to change the names of the global variables. And I have put some wordspaces in the program for better readability purposes. I Have put the ‘Landscape’ as a top view. Removed the ambient light and replaced it by two spotlights. A blue and a red light. I also removed all wild hills and valleys until I was left with a kind of creasy paper.
M_1_4_01_GDV_01

I increased the tile count to 6000. And than it gives me a kind of Perlin noise. Which might be Perlin noise indeed because the noise function is used. Let’s double that to 12000. That takes 90 seconds to render. And it makes a lot of Perlin noise. Let’s see what TileSaver does with that. The console says ‘TileSaver: 4 tilesResolution: 3200×3200. But there is nothing in my sketchfolder. Than it continues with 006.25% completed. 1/16 images saved. 012.50% completed. 2/16 images saved. 018.75% completed. 3/16 images saved. 025.00% completed. 4/16 images saved. 031.25% completed. 5/16 images saved. 037.50% completed. 6/16 images saved. 043.75% completed. 7/16 images saved. 050.00% completed. 8/16 images saved. 056.25% completed. 9/16 images saved. 062.50% completed. 10/16 images saved. 068.75% completed. 11/16 images saved. 075.00% completed. 12/16 images saved. 081.25% completed. 13/16 images saved. 087.50% completed. 14/16 images saved. 093.75% completed. 15/16 images saved. 100.00% completed. 16/16 images saved. Save: 150106_145725_3200x3200.png Done tiling. But there are still no images left in my sketch folder. It only saved 1 image of 800 x 800 because I pressed the s-key.
M_1_4_01_GDV_02

I increased the octaves amount. And added a green light to it. Tried TileSaver again. This time it works fine. It makes 1 image in 3200 x 3200 pixels by rendering 16 separate parts of the display window.
M_1_4_01_GDV_03

Used 4 lights now. A magenta, cyan, red and orange light (which is in fact yellow).
M_1_4_01_GDV_04

I have cranked up the mesh Z-Scale to 1000.
M_1_4_01_GDV_05

Lowered the NoiseFalloff to 0.01.
M_1_4_01_GDV_06

Increased z_Scale to 2000. And tilecount to a 100. And than it renders architecture like images.
M_1_4_01_GDV_07

Lowered the TileCount to 3.
M_1_4_01_GDV_08

Tile count 6 and z_Scale are now on 2000.
M_1_4_01_GDV_09

This all went very well in the last few hours. So I start once again and maybe I can do even more later.
M_1_4_01_GDV_10

Now I have made 10 variations I can start from the beginning. So I open M_1_4_01_GDV_01 and check whether I can see new ways to interpret this sketch. I have moved the hue lights to the rights side of the square. Maybe it good to increase the Z_Scale to 50. Increased the OctavesAmount to 40. And Processing is reacting very slow. Even the refreshing of the display window takes at least 15 seconds.
M_1_4_01_GDV_11

A TileCount of 12000 is of course ridicules. It takes at least more than 90 seconds to render one frame. And the result is not very good. So I lowered that to 1000. OctavesAmount is now also 100. I am going to put two more  lights in it. It seems that the distance of the light to the surface does make a difference. Closer to the surface makes it harsh. Further away from the surface gives soft shadows. Is that true? Let’s check that. All lights have now the same direction angle and concentration. Still not sure how the direction works. Anyway I made all settings for all lights the same. Except for the positions. They are all located on the corners. I did a few changes. All lights are now on the zero-Z position. Going to put them on 100 in the Z-range. Another one will be in the 200-z range. And it seems that I have to increase the concentration when the lights are higher.
M_1_4_01_GDV_12

Imported the light settings of the previous sketch into M_1_4_01_GDV_03. Saved that as M_1_4_01_GDV_13. Surprisingly it works well. We have a kind of sponge-like structure. And the lights are giving a very well representation of that structure. So what happens if I put the lights even further away. Let’s say the double the distance. That gives an interesting mix but I would like to see more black in the middle.
M_1_4_01_GDV_13

What about putting a white light in the middle? interesting. Can I make a loop with lights? I’ll give it a try. I get the following sentence in the message area: ‘java.lang.RuntimeException: java.lang.RuntimeException: can only create 8 lights’. That is weird. Thought it might be the loop itself that creates this problem. So I copy-pasted 5 lights to get 10 lights and I still get the same error message. Ok… I found a OpenGL FAQ: https://www.opengl.org/wiki/FAQ#Why_limit_to_8_lights.3F See FAQ number 29. So there you have it. It is not possible to use more than 8 lights. Hey… but you can use shaders. Fine. But I will not use them now. Did not find the white light in the middle an improvement anyway. Here is another fine discussion: https://processing.org/discourse/beta/num_1118343753.html
M_1_4_01_GDV_14

When opening this one my lights were probably in the surface. I had cranked up the Z-Scale to 200. So the light were on but hidden in the structure itself. I have put them a little higher about 500 pixels? Is this measured in pixels?
M_1_4_01_GDV_15

Variations on M_1_4_01_GDV_06
M_1_4_01_GDV_16

TileCount 1000 and Z_Scale 2000. Light falloff .9, 0, 0.
M_1_4_01_GDV_17

It seems that you can scroll with your mouse or pen on the display window to get other variations. Making variations has never been so easy.
M_1_4_01_GDV_18

The display window is divided into halves. Maybe it has something to do with the fact I only have 6 tiles. I’ll make it 9. Ok… its gone.
M_1_4_01_GDV_19

Variations on M_1_4_01_GDV_10
M_1_4_01_GDV_20

This chapter had ‘Noisy landscapes’ as a title although I did not make any landscape with it. Instead I looked for other things. And I have to admit that I found some usefull approaches which I never had used when I started with this chapter. You never know where you end up when programming. And I see that as a good thing.